If your system is not monitored with metrics but generates traces, Charmed Tempo HA can be configured to generate metrics from them and forward the metrics to a Prometheus data source using the Prometheus remote write protocol.
We are going to assume that you have already deployed Charmed Tempo HA. See this doc for how to do that.
Also see:
Deploy a worker with the metrics-generator
role
To enable trace metrics, you first need to ensure that your deployment has the metrics-generator
role enabled. How to do that depends on your specific set-up.
Monolithic mode
If you intend to use Charmed Tempo HA in its monolithic mode, you will have a worker node with the role
set to "all"
. This will implicitly also enable the metrics-generator
role, so you don’t need to do anything else to enable it, and you can skip ahead to the next section.
Distributed mode
If you intend to use Charmed Tempo HA in its distributed mode, you will need to add a worker node with the metrics-generator
role assigned.
For example:
juju deploy -m mymodel tempo-worker-k8s --config role-metrics-generator=true --config role-all=false tempo-metrics-generator
As Charmed Tempo HA workers only support a single role per application, don’t forget to unassign the "all"
role, which is enabled by default!
Then relate it to the coordinator:
juju relate -m mymodel tempo tempo-metrics-generator
Integrate with a prometheus remote-write endpoint
At this point you will have a tempo
application (an instance of the tempo-coordinator-k8s
charm) related to a worker with the metrics-generator
role enabled (or "all"
).
The tempo
application will now set active
status with a message warning you that Tempo is capable of generating metrics, but it isn’t related to a loki-push-api endpoint and therefore it isn’t generating any.
If you have deployed a dedicated worker with the metrics-generator
role, that application will go to blocked
until you relate the coordinator with a prometheus remote-write endpoint.
To solve this situation, integrate tempo
with a prometheus_remote_write provider such as the prometheus-k8s
charm. Assuming a prometheus
is deployed in the same model as tempo
:
juju relate -m mymodel tempo:send-remote-write prometheus:receive-remote-write
At this point, tempo
should be sending trace-generated metrics to Prometheus and you should be able to find them in Grafana.