Pyroscope accepts http traffic (e.g. to serve the UI) and grpc traffic (to ingest profiles) on the same port (4040).
We couldn’t find a clean way to implement the same behaviour using traefik’s ingress or ingress-per-unit primitives, and nginx didn’t seem to support this nicely either.
Therefore, we chose to split the incoming traffic into two streams: an http route for http requests, and a grpc route for all grpc traffic. The nginx workload deployed by the coordinator charm will proxy_pass/grpc_pass both streams to the same port on the worker node(s).
This also leaves us free to, in the future, expose them on different ingresses.
Therefore, we use the following routing strategy. For grpc traffic:
ingress(traefik):42424 --> coordinator(nginx):42424 --> worker:4040
And for http traffic:
ingress(traefik)/prefix --> coordinator(nginx):8080 --> worker:4040
Where the prefix is <model-name>-<coordinator-app-name>.
Example
So if you have a deployment like:
Unit Workload Agent Address Ports Message
graf/0* active idle 10.1.232.148
minio/0* active idle 10.1.232.177 9000-9001/TCP
pyroscope-worker/0* active idle 10.1.232.155 (all roles) ready.
pyroscope/0* active idle 10.1.232.153 Degraded.
s3-integrator/0* active idle 10.1.232.176
trfk/0* active idle 10.1.232.184 Serving at 10.64.140.43
and you want to send grpc profiles to pyroscope, you can do so:
- directly to the worker, at
10.1.232.155:4040 - via the coordinator, at
10.1.232.153:42424 - over traefik, at
10.64.140.43:42424
and if you want to access the pyroscope UI, you can browse:
- directly to the worker, at
10.1.232.155:4040 - via the coordinator, at
10.1.232.153:8080 - over traefik, at
10.64.140.43:8080