[community-workshop] Learn how to integrate your charm with COS-lite all the way

Tuesday, 27 juni · 15:30–16:30

Timezone: Europe/Berlin

Link to meeting: https://meet.google.com/vgr-ayek-osc


In this workshop, the Dwellir team will go through a complete deploy of COS-lite monitoring stack and show how to integrate it with the observed vm-charm as reference.

This is our current take on putting a full monitoring stack up in our own environment and hopefully will help you get started integrating your own charms (vm and/or k8).

The workshop will cover in summary:

  • Deployment of a COS-lite on microk8 (kubernetes).
  • Using cross-model relations with the VM charm observed.
  • How grafana-dashboards, prometheus (metrics), loki (logs), prometheus alert rules are added automatically by the observed charm.
  • How to ship alarms/alerts to PagerDuty and/or Slack (webhooks).
  • Room for discussions.

Our team has been working for months to get all the details right about how to use COS with our VM charms and how to integrate our own charms, so we are absolutely stoked about sharing the work with the community.

Massive attributions to the various people that have helped us this far as we now prepare to go into full production with this. @jamesbeedy @0x12b @tmihoc @marcus @jose @awnns @ppasotti


Awesome workshop, @marcus and @erik-lonroth!


Thanx alot James, we have put in alot of effort getting to a point where we feel confident enough to bring it into production. Although we know there are still some rough ends to it - we feel it is worth it because of what we hope to get back from it.

@alesstimec @jose


This is a summary of what we talked about on the workshop Learn how to integrate your charm with COS-lite all the way

How to setup a COS-lite environment and start tinkering with it

Notice that this is an example setup to encourage people to start tinkering with COS-lite
Same guide can be found at this site which include embedded player.

Creating a bridge for COS-lite

We started of by creating a bridge in netplan to separate the network for COS-lite.

Edit /etc/netplan/network-configuration-file.yaml

  version: 2
  renderer: NetworkManager
      dhcp4: true
      dhcp6: true
      - wlp9s0
      - to:
  dhcp4: false
  dhcp6: false

Install microk8s cloud

After that we continued with installing microk8s from snap


Commands used in video

Metallb is a load balancer and your gateway in to traefik. Traefik then points to services inside kubernetes. There is 3 ways to set IP-addresses, examples: 1. Static-IP:, 2. Subnet:, 3. IP-range:

$ sudo snap install microk8s --classic
$ microk8s.enable dns hostpath-storage metallb:

Bootstrap a controller on microk8s cloud

Next, we bootstrapp a juju controller on the microk8s cloud asciicast

Commands used in video

$ juju bootstrap microk8s <controller-name>

If you instead want to add the microk8s cloud to an already deployed controller use this commnads:

$ microk8s.config | juju add-k8s <cloud-name> --controller <controller-name>

Deploy COS-lite

After bootstrapping a new controller, we add a new model for COS-lite and then deploying COS stack inside that.


Commands used in video

$ juju add-model <model-name>
$ juju deploy cos-lite --trust --channel edge

To deploy COS-lite using overlays (pre-defined configurations), we can add --overlay to the command when deploying. Here are some overlay examples


Commands used in video

$ juju add-model <model-name>
$ juju deploy cos-lite --trust --channel edge --overlay ./path-to-overlay-file.yaml --overlay ./path-to-second-overlay-file.yaml

Connect to COS-lite services

Traefik is a reverse proxy, it get a gateway address from the metallb, which we can use to connect to grafana, prometheus and alertmanager. We can connect to the services by using the IP-address followed by model-name and application-name like this I’ve noticed that some applications require adding unit number as well (ex. cos-prometheus-0).

Login to grafana

We can use actions to get the admin password to be able to login to grafana.

juju run-action grafana/0 get-admin-password --wait

Make offers needed for grafana-agent

After having COS-lite deployed and ready, we can make offers of the endpoints needed for grafana-agent. Grafana-agent needs to be related to 3 endpoints to get out of it’s blocked state.


Commands used in video

$ juju offer grafana:grafana-dashboard [<offer-name>]
$ juju offer loki:logging [<offer-name>]
$ juju offer prometheus:receive-remote-write [<offer-name>]
$ juju status

Build and deploy Observed charm

Observed is built by @erik-lonroth as a reference charm to learn what COS-lite can do. It ships with a dashboard, prometheus & loki alert rule. To be able to build it you need a clone of the repo and charmcraft. After building it we can deploy observed to a new model.


Commands used in video

$ juju add-model observed
$ juju deploy ./observed.charm

Deploy and integrate grafana-agent

For observed to be able to send data to the COS-lite, we need to deploy are “bridge” between COS-lite and the charms. Grafana-agent is a subordinate charm that scrapes metrics, logs and dashboards and sends it to COS stack. Charms need to support grafana-agent integrations for it to work, which observed does, so lets deploy and relate grafana-agent right now.


Commmands used in video

$ juju deploy grafana-agent --channel edge
$ juju relate grafana-agent observed
$ juju status --relations

Consuming and relating offers

We can now see that grafana-agent is deployed under observed but is showing a blocked status. To get out of the blocked status we need to consume the offers we made previously and relate those to grafana-agent.


Commmands used in video

$ juju find-offers <controller-name>:
$ juju consume [<controller-name>]:admin/cos.grafana-dashboard [<saas-name>]
$ juju consume [<controller-name>]:admin/cos.loki-logging [<saas-name>]
$ juju consume [<controller-name>]:admin/cos.prometheus-rrw [<saas-name>]
$ juju status
$ juju relate grafana-agent grafana-dashboard
$ juju relate grafana-agent loki-logging
$ juju relate grafana-agent prometheus-rrw

Try out your new setup

After previous section it should now be possible to find new dashboards and alert rules in grafana. There should also be a dashboard called Observed Microsample. To trigger alert rules we can start to call observed charms api.

Curl Observed

$ curl http://<IP-ADDRESS-FOR-OBSERVED>:8080 # to return online
$ curl http://<IP-ADDRESS_FOR_OBSERVED>:8080/whatever # to return 404

# Example to trigger alerts for invalid api calls
$ while 1>0; do curl; sleep 0.5; done

Integrate alertmanager with other services

In the observed repo you can find examples configurations for alertmanager that can enable integration with other services like slack and pagerduty to receive notifications. Use following command to push a configuration file to alertmanager.

$ juju config alertmanager config_file=@/path/to/file.yaml

To show or check current configuration:

$ juju run-action alertmanager/0 show-config
$ juju run-action alertmanager/0 check-config