Cos Alerter
COS Alerter is a standalone webapp that is intended to make sure your COS deployment has not stopped alerting. Alertmanager is configured to continuously send alerts to COS Alerter and if it stops sending, COS Alerter notifies configured recipients.
Setup
Configure Alertmanager
The Alertmanager routes must be set up in order to deliver alerts to COS Alerter
receivers:
...
- name: cos-alerter
webhook_configs:
- url: http://<cos-alerter-address>:8080/alive?clientid=<clientid>
route:
...
routes:
...
- matchers:
- alertname = Watchdog
receiver: cos-alerter
group_wait: 0s
group_interval: 1m
repeat_interval: 1m
Make sure that group_wait is set to 0s or your alerts will be delayed.
That’s it. If you are using COS, the “Watchdog” alert rule has already been added to Prometheus.
Configure COS Alerter
Cos alerter has its own config file which must be configured in order for it to work. Here is a sample of what the config file can look like:
watch:
# The time that needs to pass without receiving an alert after which Alertmanager is considered down.
# This should be longer than the repeat_interval set in Alertmanager.
down_interval: "5m"
# When set to true, Alertmanager will not be considered down until it has received at least one alert.
# This allows you to configure COS Alerter before configuring Alertmanager.
wait_for_first_connection: true
# The list of Alertmanager instances we are monitoring. Alertmanager instances should be
# configured with the clientid=<client> parameter.
clients:
- "client0"
- "client1"
notify:
# Destinations are any [Apprise](https://github.com/caronc/apprise) compatible service string.
# See:
# https://github.com/caronc/apprise/wiki/
# https://github.com/caronc/apprise/wiki/Notify_email
destinations:
- 'mailtos://user:pass@domain/?to=example-0@example.com,example-1@example.com'
- 'slack://xoxb-1234-1234-4ddbc191d40ee098cbaae6f3523ada2d/#general'
# When Alertmanager is down, the amount of time between notifications.
repeat_interval: "1h"
# The logging level of COS Alerter
# Levels available: critical, error, warning, info, debug
log_level: "info"
Write that file to /etc/cos-alerter.yaml and you are ready.
Running COS Alerter
The easiest way to run COS Alerter is to use docker. Run with:
docker run -p 8080:8080 --mount type=bind,source="$(pwd)"/cos-alerter.yaml,target=/etc/cos-alerter.yaml,readonly -it ghcr.io/canonical/cos-alerter:latest