sunsetting `tempo-k8s`, welcome Charmed Tempo HA

charm Observability
,
1

The tempo-k8s charm, a monolithic, development product which we worked on for about one year to have a platform to test distributed tracing, is being deprecated in favour of its HA version.

The replacement is called Charmed Tempo HA and is a resilient, scalable and highly available solution for deploying and operating Tempo on kubernetes.

Migration guide

The Charmed Tempo HA solution consists of two charms, the “coordinator” and one or more “workers”. The workers are the units that actually run the Tempo backend processes; the coordinator is responsible for reverse-proxying, distributing the configuration among the workers, and load-balancing.

Also see: COS-Lite docs - Managing deployments of COS-Lite HA addons

API-wise, the coordinator charm plays the role tempo-k8s used to play. It supports the same interfaces and relation endpoints, therefore you should be able to swap tempo-k8s with the coordinator app and obtain a functionally equivalent deployment.

So once you have deployed Charmed Tempo HA (see this tutorial for instructions), you can destroy your tempo-k8s deployment and plug the coordinator in its place by relating to all tracing protocol requirers and traefik, certificates providers, COS Lite, and any other integration tempo-k8s used to have.

Data migration

tempo-k8s used an in-memory storage, whereas Charmed Tempo HA uses s3 buckets to store trace data.

It’s currently impossible to migrate data between the two, so we advise, if you are migrating a live deployment, to deploy both stacks alongside one another and wipe the old one after a sufficient amount of time has passed, to allow old history to flush itself away to irrelevance and recent history to pool up in the more robust HA solution.

Charm library migration

If you own a charm that integrates with tempo-k8s over tracing, you will need to swap the tracing and charm_tracing charm libs with their new versions who live under the tempo-coordinator-k8s namespace.

Unless you are doing something exceptionally hacky with the libraries, you should be able to follow this simple script for replacing lib.charms.tempo_k8s with lib.charms.tempo_coordinator_k8s

# in your charm repo:
# remove the old charm libs
rm -rf ./lib/charms/tempo_k8s
# fetch the new charm libs
charmcraft fetch-lib charms.tempo_coordinator_k8s.v0.charm_tracing 
charmcraft fetch-lib charms.tempo_coordinator_k8s.v0.tracing 
# replace the import paths with the new ones
find ./src ./tests \( -type d -name .git -prune \) -o -type f -print0 | xargs -0 sed -i -e 's/tempo_k8s.v1/tempo_coordinator_k8s.v0/g' -e 's/tempo_k8s.v2/tempo_coordinator_k8s.v0/g'

The APIs and public symbols have not changed, so this somewhat crude strategy should suffice for most use cases.

For examples of migrations carried out in our team, see:

However, if you haven’t done so already, this is a perfect opportunity to start using the charm_tracing_config helper function from the tracing library to help you configure charm tracing in the right way!

Using the charm_tracing_config helper

Old code:

from charms.tempo_coordinator_k8s.v0.charm_tracing import trace_charm
from charms.tempo_coordinator_k8s.v0.tracing import TracingEndpointRequirer

@trace_charm("tracing_endpoint",  "tracing_cert")
class MyCharm(CharmBase):
    def __init__(...):
        ...
        self.tracing = TracingEndpointRequirer(self)

    def tracing_endpoint(self):
        # actually some more complex logic here, see below.
        return self.tracing.get_endpoint("otlp_http")

    def tracing_cert(self):
        return "/path/to/cert.crt"

In earlier revisions of the tracing libraries, your charm would be responsible for passing to the charm tracing machinery the right combination of endpoint and cert depending on the situation:

  • endpoint is https
    • cert is not on disk: tracing disabled (or flushing would fail with TLS errors)
    • cert is on disk: tracing enabled
  • endpoint is http: tracing enabled regardless of cert

In order to disable charm tracing, the caller would be responsible for returning None from the tracing endpoint passed to trace_charm. This was error-prone. Consequently we have written a little helper function to give trace_charm the right combination of required parameters given a TracingEndpointRequirer parameter and the expected location of a tls certificate. All the charm is responsible for, now, is to ensure that a CA cert is written to that location as soon as it becomes available. Note that that CA cert must be the same one that Tempo is trusting; otherwise things will not work.

New code:

from charms.tempo_coordinator_k8s.v0.charm_tracing import trace_charm
from charms.tempo_coordinator_k8s.v0.tracing import TracingEndpointRequirer, charm_tracing_config

@trace_charm("tracing_endpoint",  "tracing_cert")
class MyCharm(CharmBase):
    def __init__(...):
        ...
        self.tracing = TracingEndpointRequirer(self)
        self.tracing_endpoint, self.tracing_cert = charm_tracing_config(self.tracing, "/path/to/cert.crt")
2 Likes