In Grafana Tempo, a service graph is a visual representation of the interrelationships between various services. Service graphs help you to understand the structure of a distributed system, and the connections and dependencies between its components.
See more: Service graphs
Service graphs are generated in charmed tempo HA and pushed to a metrics storage (charmed prometheus or charmed mimir HA). Then, they can be represented in charmed grafana as a graph. So, you need the following integrations to fully use service graphs:
send-remote-writegrafana-sourcereceive-datasource
Deploy Tempo HA
Check this guide to deploy Tempo HA.
If you intend to deploy charmed Tempo HA in its distributed mode, make sure to deploy a dedicated worker with a metrics-generator role using the config option role-metrics-generator=true.
By default, deploying charmed Tempo HA in its monolithic mode with role all, will have metrics-generator enabled.
Integrate over send-remote-write
The way service graphs work is that the metrics-generator, a Tempo component that derives metrics from ingested traces, processes traces and generates service graphs in the form of Prometheus metrics and pushes them to prometheus/mimir remote write endpoints.
See more: How to enable trace metrics
We’ll be using charmed prometheus as the metrics backend, but charmed mimir HA can be used interchangeably as well.
juju deploy prometheus-k8s prometheus --channel edge --trust
juju integrate tempo:send-remote-write prometheus
Integrate over grafana-source
For service graphs to be represented in Grafana, we need to configure tempo and prometheus as datasources in the same grafana instance.
juju deploy grafana-k8s grafana --channel=edge --trust
juju integrate tempo:grafana-source grafana
juju integrate prometheus:grafana-source grafana
Integrate over receive-datasource
What is left is to configure Tempo’s datasource to link to Prometheus’s datasource and that can be achieved by exchanging datasource UIDs using the grafana_datasource_exchange interface.
juju integrate tempo:receive-datasource prometheus:send-datasource
See more: grafana_datasource_exchange
View service graphs in Grafana
Open Grafana web UI form your local web browser and login using the admin password.
Then, Toggle Menu → Explore → Choose your Tempo datasource → Service Graph → Run query
This should result in a graph similar to the one below:
Service graphs work by inspecting traces and looking for spans with parent-children relationship that represent a request. So, the nodes and edges that would come up in the graph depend on the workload traces generated by your charmed applications.
Nodes → represent the charmed applications emitting workload spans.
Edges → represent a workload request flow from charmed application A to charmed application B given that their workloads generate spans using the same tracing protocol (e.g: OpenTelemetry protocol).