Loki-coordinator-k8s docs - distributed deployment

charm
, , ,
1

Table of Contents

This PR uses the abstraction of the coordinator class in the Loki Coordinator charm. In tandem with: implementation of generic worker by Abuelodelanada · Pull Request #8 · canonical/loki-worker-k8s-operator · GitHub

Table of Contents

Testing Instructions

“Monolithic Mode”: (coordinator + worker + s3 + grafana + flog)

  1. Deploy the charms: in a loki model:
    • Worker: juju deploy loki-worker-k8s worker --channel=edge --config role-all=True --trust
    • Loki coord: juju deploy ./*.charm loki --resource nginx-image=ubuntu/nginx:1.18-22.04_beta --resource nginx-prometheus-exporter-image=nginx/nginx-prometheus-exporter:1.1.0 --trust
  • Minio: juju deploy minio --channel=stable --trust --config access-key="AccessKey" --config secret-key="SecretKey"
  • S3_integrator: juju deploy s3-integrator s3 --channel stable --trust
    • Sync S3 credentials: juju run s3/leader sync-s3-credentials access-key=AccessKey secret-key=SecretKey
    • Configure endpoint: juju config s3 endpoint="http://minio-0.minio-endpoints.loki.svc.cluster.local:9000"
  • Grafana: juju deploy grafana-k8s grafana --channel edge --trust
  • Flog: juju deploy flog-k8s flog --channel edge
  1. Log into the Minio console: http://MINIO_IP:9001 with user: AccessKey and Password: SecretKey and create a bucket called loki
  2. Relate all the charms:
    • juju relate loki s3
    • juju relate loki grafana:grafana-source
    • juju relate loki worker
    • juju relate loki flog

Now you can verify using Grafana that Flog is sending log to Loki Monolithic deployment.

Verify logs are persisted in S3 by replacing the worker unit

  1. Deploy a new worker2: juju deploy loki-worker-k8s worker2 --channel=edge --config role-write=True --trust
  2. Remove the relation between loki and worker: juju remove-relation loki worker
  3. Create the relation between loki and worker2: juju relate loki worker2
  4. Verify in Grafana logs are still there:

image

“Distributed mode”: (coordinator + read + work + backend + s3 + grafana + flog)

  1. Deploy the charms: in a loki model:

    • Read: juju deploy loki-worker-k8s read --channel=edge --config role-read=True --trust
    • Write: juju deploy loki-worker-k8s write --channel=edge --config role-write=True --trust
    • Backend: juju deploy loki-worker-k8s backend --channel=edge --config role-backend=True --trust
    • Loki coord: juju deploy ./*.charm loki --resource nginx-image=ubuntu/nginx:1.18-22.04_beta --resource nginx-prometheus-exporter-image=nginx/nginx-prometheus-exporter:1.1.0 --trust
    • Minio: juju deploy minio --channel=stable --trust --config access-key="AccessKey" --config secret-key="SecretKey"
    • S3_integrator: juju deploy s3-integrator s3 --channel stable --trust
      • Sync S3 credentials: juju run s3/leader sync-s3-credentials access-key=AccessKey secret-key=SecretKey
      • Configure endpoint: juju config s3 endpoint="http://minio-0.minio-endpoints.loki.svc.cluster.local:9000"
    • Grafana: juju deploy grafana-k8s grafana --channel edge --trust
    • Flog: juju deploy flog-k8s flog --channel edge

  2. Log into the Minio console: http://MINIO_IP:9001 with user: AccessKey and Password: SecretKey and create a bucket called loki

  3. Relate all the charms:

    • juju relate loki s3
    • juju relate loki grafana:grafana-source
    • juju relate loki write
    • juju relate loki read
    • juju relate loki backend
    • juju relate loki flog

Now you can verify using Grafana that Flog is sending log to Loki Distributed deployment.

Verify logs are persisted in S3 by replacing the write unit

  1. Deploy a new write2: juju deploy loki-worker-k8s write2 --channel=edge --config role-write=True --trust
  2. Remove the relation between loki and write: juju remove-relation loki write
  3. Create the relation between loki and write2: juju relate loki write2
  4. Verify in Grafana logs are still there:

image

If you have an existing COS-Lite deployment and you wish to modify the Loki application to a distributed, highly-available application, follow these steps.

Deploy loki-coordinator-k8s and loki-worker-k8s

In the same juju model as you have COS-Lite deployed, remove the existing Loki application.

juju remove-application loki

Once the existing Loki application is removed, deploy the loki-coordinator-k8s app.

juju deploy loki-coordinator-k8s --channel edge loki --trust

We recommend naming your loki-coordinator app "loki" as that is the single entry point to the Loki deployment as a whole: you will never have to interact with the worker nodes directly.

Deploy loki-worker-k8s

juju deploy loki-worker-k8s --channel edge loki-worker --config role-all=True --trust

In this tutorial we deploy the ‘monolithic’ version of loki, where a single worker node is assigned all the roles. For alternative deployment modes and a migration guide, refer to this other doc.

Integrate with s3

Loki uses object storage for storing logs and the charm consequently requires an s3 integration.

If you don’t have an s3 bucket ready at hand, follow this guide to deploy minio in your testing environment.

Once you’re done deploying minio and s3, you can run:

juju integrate loki s3

Integrate coordinator and workers

The coordinator and the workers need to be related.

juju integrate loki loki-worker

Wait for loki to go to active/idle. At this point your juju status should look like this:

image
image740×178 48.8 KB

Coordinator is reporting ‘Degraded’ because the number of worker units are less than the recommended amount (see this open bug for progress on making the status message more informative).

Integrate with COS-Lite

You can enable self-monitoring for loki by integrating it with the other COS-Lite components.

juju integrate s3:s3-credentials loki:s3
juju integrate loki:grafana-dashboard grafana:grafana-dashboard
juju integrate loki:grafana-source grafana:grafana-source
juju integrate loki:metrics-endpoint prometheus:metrics-endpoint
juju integrate loki:tempo-cluster tempo-worker:tempo-cluster
juju integrate traefik:traefik-route loki:ingress

Integrate with a CA

If you have a charm offering a certificates endpoint such as self-signed-certificates, you can integrate it with loki:

juju integrate loki:certificates ca:certificates

to enable logs to be sent to loki over https (or grpcs).

Note that for this to work, loki needs to trust the same CA that traefik trusts. If you’re using different CAs to provide certificates to loki and traefik, you’ll need to integrate the CA charms with cert-transfer.