Many COS HA charms, such as Tempo, Loki, and Mimir, follow a “Coordinator/Workers”. This pattern is described in detail here. The following post extends this description, focusing on how telemetry is collected from Coordinator/Worker charms.
Metrics
- Metrics configuration is provided by the Coordinator to a metrics scraper via the
metrics-endpointrelation endpoint on theprometheus_scrapeinterface. The metrics configuration provided includes all workloads (the Coordinator’s nginx and all Worker workloads) - Coordinator:
- uses
MetricsEndpointProviderto send alert rules to the scraper - uses
MetricsEndpointProviderto send scrape targets for all workloads in the coordinated-worker solution (nginx and all worker workloads) - nginx workload’s metrics are scraped by the scraper
- uses
- Worker:
- no metrics configuration occurs in the Worker charm
- workload metrics are scraped by the scraper
GETrequests direct to every Worker unit
Relation topology:
Data flow:
Logs
- Logging configuration is provided to the Coordinator via the
loggingrelation endpoint using theloki_push_apiinterface - Coordinator:
- uses
LogForwarderto configure pebble to send all coordinator workload (nginx) logs to the logging store - extracts logging configuration by interrogating the
loggingrelation data directly (usingloki_endpoints_by_unit, not by using a library from loki) and injects that into theClusterrelation
- uses
- Worker:
- uses
ManualLogForwarder(a local version similar toLogForwarder) to configure pebble to send all Worker workload logs to the logging store, using configuration from theClusterrelation
- uses
Relation topology:
Data flow:
Traces
- Tracing configuration is provided to the Coordinator via the
charm-tracingandworkload-tracingrelation endpoints, both using thetracinginterfacecharm-tracingin the Coordinator is used to configure tracing in all coordinated-worker charms (Coordinator and all related Workers)
- Coordinator:
- gets tracing configuration from
charm-tracingandworkload-tracingby instantiating aTracingEndpointRequirerfor each - sends the protocols required for
charm-tracingandworkload-tracingback to the related tracing provider using the instantiatedTracingEndpointRequirerobjects - uses the
charm-tracingconfiguration to:- configure the Coordinator charm to emit traces using ops.tracing machinery
- forward
charm-tracingconfiguration to the Workers via theClusterdatabag
- uses the
workload-tracingconfiguration to:- forward
workload-tracingconfiguration to the Workers via theClusterrelation
- forward
- gets tracing configuration from
- Worker:
- gets tracing configuration from the
Clusterrelation - configures the workload to emit traces to the provided tracing store
- gets tracing configuration from the
Relation topology:
Data flow:





