COS aka "the new observability stack"

charm
1

Hi,

I discovered recently that the LMA stack was in the process of being replaced by the new “COS” observability stack (https://juju.is/docs/lma2) which I’m really excited about because LMA seemed way too outdated to me (really, Nagios ? Greylog ?) COS sounds just great and I must admit that it perfectly fits our needs since we’re already using a very similar homemade stack based on most of the same components (Grafana, Prometheus, Loki, AlertManager, …). I still have a few questions though … Right now, building such stack relies on many components that still miss on Charmhub, especially some exporters that are needed such as a more complete Openstack exporter (I already asked for that in another topic), the node, haproxy, mysql, rabbitmq, … exporters and maybe some more. Most of these exporters already exists aned are doing well but as far as I can see, most of them are not packaged as Charms. Is it something that is beeing worked on currently ?

As for COS itself … looking at the blog post, it mostly talks about Kubernetes but what about Openstack ? As I said, I’m currently using this kind of stack for our old Openstack clusters (that are not Charm based) and this is a really great addition. I would love to have COS targeted at Openstack too.

And regarding the bundle itself, I see you’re using Karma as the alert UI. May I advise you to take a look and consider Alerta (https://github.com/alerta/alerta) which is a nice alerting UI with a more traditionnal approach than Karma that we evaluated but found way too complicated especially in our case where we have people dedicated to 24/7 supervision with big wall screens, filter based display and too much floating elements was a no go for them. Alerta has a more classical approach which they liked best.

May I also advise to take a look to the “Exporter Exporter” (https://github.com/QubitProducts/exporter_exporter) which is a reverse proxy for exporters so that you can expose only 1 port and have as many exporters you need on a host. Very usefull when, like in our case, very strict regulations on open ports or if you want things like SSL based communication between Prometheus and exporters even if the exporter itself don’t support SSL.

Well, I’m really excited about this stack and I’d be happy to help in building a really nice observability platform.

Best regards.

2 Likes
2

Oh hello there!

Have a look at Deploy Canonical Observability Stack Lite using Charmhub - The Open Operator Collection . All the pieces are falling in place, but since we have not yet released a stable version, they are not searchable yet :wink: Documentation is spare so far, but lives under Juju | Canonical Observability Stack. (We initially named COS As LMA2, we have a few link migrations to perform.)

Karma is not an integral part of COS. We are planning to use Grafana itself to display alerts (and we will migrate the Grafana K8s charm to Grafana 8 to achieve that). Karma is something we charmed because colleagues in Canonical have use-cases to monitor alerts from many different COS installations, and Karma fit the bill for that nicely.

COS itself runs on K8s, but we are designing it and engineering it so that it will be capable of monitoring all Canonical software eventually. Juju makes our work pretty easy like that :slight_smile: We also have already some significant backwards compatibility with LMA via Deploy Canonical Observability Stack Proxy using Charmhub - The Open Operator Collection, and we plan to do more in the next weeks and months (we miss a couple Prometheus relation interfaces, and logging).

In terms of aggregating exporters, I think in many cases that will not be needed. We already charmed Grafana Agent for K8s and we will do it as well as machine charm. Grafana agent will scrape locally, and push data into COS with remote-write, effectively getting rid of any open inbound ports (at least in terms of monitoring) for your machines.

P.S. We intend to join the Community Hour and present our work and plans about COS. The slot is not set yet, but it will be set early next week, I hope :wink:

3

So, it is official: we are organizing a COS Lite - Sneak preview community workshop on Friday, February 25th :slight_smile:

1 Like