Grafana dashboard does not reappear in grafana after shutdown

getting started
,
#1

Hi, We are continuously tinkering with the COS-stack and grafana-agent. One thing I noticed is that dashboards sent from grafana-agent doesn’t reappear in grafana after the computer has been of for a couple of days. Even though the relation is still maintained.

After booting the machine

image
image1198×277 19.1 KB
image
image712×322 78.4 KB

After redoing the relation

image
image1267×620 41.7 KB
image
image802×363 131 KB

Alert rules does appear. Any idea why?

#2

I’ve seen disappearing dashboards before. But I can’t remember if we debugged or fixed the issue @0x12b

1 Like
#3

Hi @marcus!

Thanks for the report! We’ll follow this in the following issue.

Thanks! Jose

2 Likes
#4

Let us know if we can help.

We are just completing work exploring COS lite by implementing a reference charm like I showed you in other threads and @marcus just managed to get loki alert rules in there now.

The final piece(s) of this puzzle for us is to understand how we can:

  1. integrate with some alerting or ticketing service, such as pagerduty or/and Zammad
  2. Add tls to be able to expose the service on internet.

From there, we should be about brave enough to go into production.

This is also complete enough to be foundation for the promised community workshop we plan to host. Deploying the complete stack.

There are however some rough edges we fear, but possibly can live with and work with you guy to get stable.

#5

Hi @erik-lonroth

About these ones:

  1. I’m thinking out loud, but Alertmanager is able to send notifications using webhooks and, as far as I understand, you can connect it with PagerDuty
  2. Traefik (COS-Lite ingress) implements the requires side of the interface tls-certificates. The provider side of this relation is implemented in 4 different charms. Maybe you can use one of these.
2 Likes