How to forward charm logs to COS

doc
, ,
1

Also see: How to instrument a charm for distributed tracing

Charm execution logging

When we talk about logging in Juju, we usually refer to logs emitted by the workload that a charm is operating. However, you can also collect the logs emitted by the charm code execution itself for debugging and troubleshooting purposes.

Fetch the charm_logging charm library

To start, grab the charm_logging lib:

charmcraft fetch-lib charms.loki_k8s.v0.charm_logging

The charm_logging lib contains all you need to configure your charm to automatically forward any logs to a Loki instance over an existing loki_push_api integration.

This howto assumes that your charm already has an integration to loki-k8s over the loki_push_api relation interface. See integrating with loki-k8s over loki_push_api for instructions. The charm_logging lib will use the same integration. This means that the same Loki instance that stores workload logs will also receive charm logs. In practice, you might want them to be separate Loki instances. To achieve that, add a separate integration to your charm and point charm_logging to that one instead.

This means that, if your charm is related to loki-k8s charm and loki-k8s is related to grafana-k8s over grafana-source, you will be able to inspect in real time from the Grafana explorer the logs emitted by your charm.

Quickstart: using the charm_logging library

The main entry point to the charm_logging library is the log_charm decorator. Assuming you already have an integration over tracing, and you’re already using lib.charms.loki_k8s.v1.loki_push_api.LokiPushApiConsumer, you will need to:

  • import lib.charms.loki_k8s.v0.charm_logging.log_charm
  • decorate your charm class (if you have multiple, only decorate the ‘last one’ i.e. the one you pass to ops.main.main()!) with log_charm.
  • pass to log_charm forward-references to:
    • the url of a loki-push-api endpoint
    • [optional] absolute path to a CA certificate on the charm container disk

Caution should be exercised in that:

  • if the loki_push_api_endpoint you pass to the decorator is not None, charm logging will be enabled
  • if charm logging is enabled, and the endpoint is HTTPS, but no cert_path is passed to the decorator, (or, the path is invalid/does not exist), you will get errors.

To help generate the right configuration and avoid inconsistent situations, loki_push_api comes with a charm_logging_config helper to determine when to disable charm logging because the endpoint is HTTPS but we don’t have a certificate on disk yet. The charm is still responsible for writing the cert file to disk as soon as one is available.

For example:

from lib.charms.loki_k8s.v0.charm_logging import log_charm
from lib.charms.loki_k8s.v1.loki_push_api import LokiPushApiConsumer, charm_logging_config

@log_charm(loki_push_api_endpoint="my_logging_endpoint", cert_path="cert_path")
class MyCharm(...):
    # if your charm has an integration to a CA certificate provider, you can copy the CA certificate on the charm container and use it to encrypt logs sent to Loki
    _cert_path = "/path/to/cert/on/charm/container/cacert.crt"

    def __init__(self, ...):
        # the tracing integration
        self.logging = LokiPushApiConsumer(self, ...)
        
        # this data will be picked up by the decorator and used to determine where to send the logs to, and what CA cert (if any) to use to encrypt them if the endpoint is https.
        self.my_logging_endpoint, self.cert_path = charm_logging_config(
            self.logging, self._cert_path)

At this point any logs emitted throughout the lifecycle of MyCharm are automatically forwarded to Loki. All that you will see in the juju debug-log is now available in grafana.

Future work

  • Associating charm traces to logs and vice versa (exemplars) in grafana
1 Like
2

this doc should be merged in loki’s diataxis tree much the same way as this doc was merged in Tempo’s.

3

WoW - this is amazing! I’ve forwarded this to our team at Dwellir! Super useful !