How to instrument your charm with tracing telemetry

[ppasotti (Pietro Pasotti)]

This howto has been incorporated in the tempo docs.

In order to instrument a charm with tracing telemetry, you will need to:

1. Set up a model with cos-lite:

juju add-model clite
juju deploy cos-lite --trust

See more: Charmhub | cos-lite

2. Deploy tempo:

juju deploy tempo-k8s

See more: Charmhub | tempo-k8s

3. Integrate tempo with cos-lite:

jhack imatrix fill

See more: jhack, GitHub | jhack > imatrix > fill

Alternatively, you can integrate manually by:

juju integrate tempo:logging loki:logging
juju integrate tempo:ingress traefik:ingress
juju integrate grafana:grafana-source tempo:grafana-source
juju integrate prometheus:metrics-endpoint, tempo:metrics-endpoint
juju integrate grafana:grafana-dashboard, tempo:grafana-dashboard
juju integrate traefik:tracing, tempo:tracing
juju integrate prometheus:tracing, tempo:tracing

At some point there will be an overlay bundle to deploy cos-lite + tempo and integrate them; see this PR to follow the progress on the overlay.

At this point you should have a working cos-lite plus tempo deployment; something like this:

4. Fetch the charm_tracing and the tracing libs:

charmcraft fetch-lib charms.tempo-k8s.v1.charm_tracing
charmcraft fetch-lib charms.tempo-k8s.v2.tracing

See more: charmcraft fetch-lib, Charmhub | tempo-k8s > Libraries > charm-tracing, Charmhub | tempo-k8s > Libraries > tracing

5. Add opentelemetry exporter dependency used by charm_tracing to your requirements.txt:

opentelemetry-exporter-otlp-proto-http>=1.21.0

6. Add an integration with tempo:

# in charmcraft.yaml
provides:
  tracing:
    interface: tracing
    limit: 1

See more: File charmcraft.yaml > provides

7. Instrument your charm code:

# in /your/charm/project/src/charm.py
from charms.tempo_k8s.v1.charm_tracing import trace_charm
from charms.tempo_k8s.v2.tracing import TracingEndpointRequirer

@trace_charm(tracing_endpoint="tracing_endpoint")
class MyCharm(CharmBase):
    def __init__(self, *args):
        super().__init__(*args)
        # add a provider wrapper for the tracing endpoint
        self.tracing = TracingEndpointRequirer(self, protocols=["otlp_http"])

    @property
    def tracing_endpoint(self) -> Optional[str]:
        """Tempo endpoint for charm tracing."""
        if self.tracing.is_ready():
            return self.tracing.get_endpoint("otlp_http")
        return None

By default, the traces generated by this charm will be tagged with service name equal to the class name of the charm, so "MyCharm" in this case. You override this default by passing service_name="foo" to trace_charm.

8. Pack, deploy, and integrate your charm with cos-lite:

charmcraft pack
juju deploy ./my-charm-operator_ubuntu-20.04-amd64.charm mycharm \
  $(yq eval '.resources | to_entries | map(select(.value.upstream-source != null) | "--resource " + .key + "=" + .value.upstream-source) | join(" ")' charmcraft.yaml)
juju integrate mycharm:tracing tempo-k8s:tracing

cos-lite is a kubernetes bundle. tempo is a Kubernetes charm.

See more: charmcraft pack, Juju | juju deploy, Juju | juju integrate

9. View the traces for the charm.

Open the Grafana dashboard in a browser (see here for more detailed instructions).

Next, navigate to the traces for your charm:

• go to Explore and select the Tempo datasource.
• pick the service_name you gave to MyCharm above (the default is the application name) to see the traces for that charm
  
• click on a trace ID to visualize it. For example, this is the trace for an update-status event on the Tempo charm itself:
  

Mapping events to traces with jhack tail

jhack tail supports a -t option to show the trace IDs associated with a charm execution:

This means that you can tail a charm, grab the trace id from tail, put it in the grafana dashboard query and get to the trace in no time.

[ghibourg (Ghislain Bourgeois)]

The instructions are missing the fetching and importing of tempo_scrape.

charmcraft fetch-lib charms.tempo_k8s.v0.tempo_scrape

```python
from charms.tempo_k8s.v0.tempo_scrape import TracingEndpointProvider

Also, the deploy command for tempo does not have the right charm file name. It should be tempo-k8s_ubuntu-22.04-amd64.charm according to my testing today.

The tempo charm also goes into error in logging-relation-changed, failing to start promtail because the port 9095 seems already in use.

I did not get the chance to debug more.

I think this looks interesting overall, I will definitely try to play with it a bit more.

[ppasotti (Pietro Pasotti)]

thanks for the feedback, will polish it up a bit today

[ppasotti (Pietro Pasotti)]

I’m locked out of the tempo charm while IS handles giving us back ownership after I transferred the repo to the Canonical org, so I can’t merge a fix for that one, but the branch is up

[mthaddon (Tom Haddon)]

@ppasotti

Integrate tempo with cos-lite:

Would it be possible to include the regular juju integrate commands here for those that might not have jhack installed (e.g. on a production environment)?

[ppasotti (Pietro Pasotti)]

good point! we have no official docs on that yet.

[ppasotti (Pietro Pasotti)]

done @mthaddon !