Forming a cluster
Alertmanager supports clustering and all you need to do to create/update a cluster is to rescale the application. This can be done in two ways.
Let’s say we have one alertmanager unit running and we want to scale the deployment to three units.
With juju add-unit we can achieve that using the --num-units argument and the number of units we want to add:
juju add-unit alertmanager-k8s --num-units 2
or using juju scale-application and the total number of units we want:
juju scale-application alertmanager-k8s 3
Regardless of which of the two options you use, juju status --relations --color will show you the status of the cluster.
Internally, HA is achieved by providing each Alertmanager instance at least one IP address of another instance. The cluster would then auto-update with subsequent changes to the units present.
Verification
Pebble plan
Cluster information is passed to Alertmanager via --cluster.peer command line arguments. This can be verified by looking at the current pebble plan:
> $ juju exec --unit alertmanager-k8s/0 -- \
PEBBLE_SOCKET=/charm/containers/alertmanager/pebble.socket \
pebble plan
services:
alertmanager:
summary: alertmanager service
startup: enabled
override: replace
command: alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager --web.listen-address=:9093 --cluster.listen-address=0.0.0.0:9094 --cluster.peer=10.1.179.220:9094 --cluster.peer=10.1.179.221:9094
HTTP API
To manually verify a cluster is indeed formed, you can query the alertmanager HTTP API directly:
> curl -s $ALERTMANAGER_IP:9093/api/v1/status \
| jq '.data.clusterStatus.peers[].address'
"10.1.179.220:9094"
"10.1.179.221:9094"
"10.1.179.217:9094"