Metadata
| Key | Value |
|---|---|
| Summary | Integrating with an Alertmanager receiver |
| Categories | integrations |
| Difficulty | 2 |
| Author | Ibrahim Awwad |
- Introduction
- Step 0: Setting Up Telegram Integration
- Step 1: Setting Up Pushover Integration
- Step 2: Setting Up PagerDuty Integration
- Step 3: Configuring Juju Alertmanager
- Step 4: Test The Receivers
Introduction
In this tutorial, we’ll guide you through the process of integrating the Juju Alertmanager Charm with external notification services. These integrations enable you to receive alerts and notifications from COS Lite directly on your preferred messaging platform. For the sake of simplicity, we’ll cover Telegram, Pushover and PagerDuty integrations, but the steps demonstrated will be fairly similar to any other receiver.
Before proceeding with the integration steps, ensure that you have COS Lite deployed in your environment. If you haven’t deployed COS Lite yet, you can follow the detailed instructions provided here.
Step 0: Setting Up Telegram Integration
To integrate Alertmanager with Telegram, you need two pieces of information: your Telegram bot API token and chat ID . These will be used to create an Alertmanager Telegram Receiver, which routes alerts to your Telegram chat.
Set Up Your Telegram Bot
- Open the Telegram app and find BotFather.
- Type
/newbotto create a new bot and follow the prompts. - Once created, you’ll receive a message containing your API token key.
Obtain the Chat ID
- Add your bot to a group chat.
- Interact with the bot by sending a dummy message starting with
/. - Send an HTTP request to your bot’s API endpoint:
https://api.telegram.org/bot<bot_api_token>/getUpdates. - Retrieve the chat ID from the response JSON.
- If the request returns an empty response, remove and re-add the bot to the group chat, then retry the request.
Step 1: Setting Up Pushover Integration
Integrating Alertmanager with Pushover requires obtaining your Pushover User Key and Token. These credentials are necessary to create an Alertmanager Pushover Receiver, enabling notifications to be sent to your Pushover account.
Obtain Pushover User Key and Token
- Sign up for a Pushover account if you haven’t already.
- Log in to your Pushover dashboard.
- Create a new application and note the generated API token.
- Retrieve your User Key from your Pushover dashboard.
Step 2: Setting Up PagerDuty Integration
Before diving into integrating Alertmanager with PagerDuty, it’s essential to grasp a few concepts within PagerDuty:
- PagerDuty Service: This is where your alerts are directed. Services have integration types; for our purposes, we’ll focus on Prometheus (using PagerDuty EventsV1) or EventsV2, offering more control over alerts.
- PagerDuty Event Orchestration: This automates the flow of incoming alerts. It allows you to set up flows for each alert, routing them to services without the need for routing through AlertManager. Advanced rules can be set, but we’ll keep it simple for this tutorial.
These concepts will guide us as we explore two integration methods: via PagerDuty service and via Event Orchestration. Each method serves different needs, catering to varying levels of automation and customization.
- Integration via PagerDuty Service: Ideal for straightforward integration when routing alerts directly from Alertmanager, or specifying the routing in Alertmanager.
- Integration via Event Orchestration: Recommended for automation and customized alert routing from PagerDuty without having to worry about further configuration in Alertmanager.
We’ll cover both methods to accommodate users with diverse requirements.
Integration via PagerDuty Service
- Navigate to the Services menu and select Service Directory.
- If adding to an existing service, click its name, then the Integrations tab, and Add a new integration. For a new service, follow the documentation for creating a new service.
- Choose Events API v2 or Prometheus from the Integration Type menu. For this tutorial, we’ll select Events API v2.
- Save the integration by clicking Add Service or Add Integration. Copy the Integration Key provided for later use.
Integration via Event Orchestration
For this method, we’ll create two PagerDuty services and utilize Event Orchestration to route alerts:
-
Create two PagerDuty services (e.g., Juju Alertmanager Global Service 1 and Juju Alertmanager Global Service 2) as explained in the previous step.
-
In the PagerDuty web app, navigate to Automation > Event Orchestration and select or create your desired orchestration.
-
Create a service route for each service. For example, route alerts containing “hola” in the event summary to the first service, and route the rest to the second service.
- Go to Integrations, copy your default Integration Key, and store it safely for later use.
Step 3: Configuring Juju Alertmanager
To configure Juju Alertmanager to work with Telegram, Pushover and PagerDuty, you’ll need to draft an alertmanager.yml configuration file and a templates.tmpl template file, replacing the placeholders with the information retrieved from previous steps.
Drafting alertmanager.yml
Below is a basic alertmanager.yml configuration file to integrate Juju Alertmanager with Telegram, Pushover, and PagerDuty. Here’s an overview of the main settings:
- Routing Configuration:
receiver: Specifies the default receiver for alerts, set to ‘telegram’.group_by: Groups alerts based on Juju model UUID, application, and model.continue: Determines whether to continue processing additional routes after a match is found.group_wait,group_interval,repeat_interval: Control grouping and repetition of alerts; unresolved alerts are sent out every 1 hour.
- Receiver Configuration:
- Configures four receivers: ‘telegram’, ‘pushover’, ‘pagerduty_service’ and ‘pagerduty_orchestration’.
- Specifies settings for Telegram, Pushover, and PagerDuty, including API tokens, chat IDs, user keys, and tokens.
global:
resolve_timeout: 5m
route:
receiver: 'telegram'
group_by:
- juju_model_uuid
- juju_application
- juju_model
continue: false
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
routes:
- receiver: 'telegram'
group_wait: 10s
match_re:
severity: critical|warning|none
continue: true
- receiver: 'pushover'
group_wait: 10s
match_re:
severity: critical|warning|none
continue: true
- receiver: 'pagerduty_service'
group_wait: 10s
match_re:
severity: critical|warning|none
continue: true
- receiver: 'pagerduty_orchestration'
group_wait: 10s
match_re:
severity: critical|warning|none
continue: true
receivers:
- name: 'telegram'
telegram_configs:
- bot_token: YOUR_TELEGRAM_BOT_TOKEN
chat_id: YOUR_CHAT_ID
- name: 'pushover'
pushover_configs:
- user_key: YOUR_PUSHOVER_USER_KEY
token: YOUR_PUSHOVER_TOKEN
- name: 'pagerduty_service'
pagerduty_configs:
- routing_key: YOUR_PAGERDUTY_ROUTING_KEY_1
- name: 'pagerduty_orchestration'
pagerduty_configs:
- routing_key: YOUR_PAGERDUTY_ROUTING_KEY_2
Alertmanager provides additional customization options for Telegram, Pushover and PagerDuty integrations. These options allow you to configure settings such as whether to notify about resolved alerts, specify an optional device to send notifications to, or adjust the frequency of repeated notifications from Pushover servers. While these advanced configurations were not included in this tutorial for simplicity, they can be easily implemented by following the alertmanager configuration documentation.
Drafting templates.tmpl
This file defines message templates for notifications sent to Telegram and Pushover. Each template includes information about the alerts, such as summary, description, severity, status, start and end times, and a URL. These templates can be customized as needed, but for the sake of simplicity in this tutorial, they are kept straightforward.
{{ define "telegram.default.message" }}
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }}
*Description:* {{ .Annotations.description }}
*Severity:* {{ .Labels.severity }}
{{ end }}
{{ end }}
{{ define "pushover.default.message" }}
{{ range .Alerts }}
Alert: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
Severity: {{ .Labels.severity }}
{{ end }}
{{ end }}
The provided templates may result in notifications that appear somewhat unattractive and lacking. This was done intentionally to illustrate how to override the default Alertmanager templates for Telegram and Pushover.
Note on PagerDuty Templates: We’ve not included a template for PagerDuty as there’s no direct support for such customization through Alertmanager templates. However, fields can be customized directly via the Alertmanager PagerDuty configuration if needed, as mentioned here.
Applying Configuration
Run the following command to apply the configuration to the Alertmanager charm:
juju config alertmanager \
config_file='@/path/to/alertmanager.yml' \
templates_file='@/path/to/templates.tmpl'
Replace /path/to/alertmanager.yml and /path/to/templates.tmpl with the actual paths to your configuration files.
Step 4: Test The Receivers
To ensure that the integration with Telegram, Pushover and PagerDuty is functioning correctly, you can utilize the existing watchdog alert that periodically fires. This alert serves as a convenient way to verify that notifications are being delivered to both platforms.
Pushover Results
Please check your Pushover app on your device to view the notification.
Telegram Results
Please check your Telegram chat on your device to view the notification.
PagerDuty Results
Please check your PagerDuty app on your device to view the notification.
When testing with the watchdog alert, expect to observe two incidents in PagerDuty:
- One incident routed to the PagerDuty service named
Juju AlertManager Direct Service, also referred to aspagerduty_service. - Another incident routed to the PagerDuty orchestration named
Juju AlertManager Global Service 2, also referred to aspagerduty_orchestration.
The third PagerDuty service, Juju AlertManager Global Service 1, won’t have any incidents triggered. This service is configured to receive alerts only if they contain the label “hola” in their summary, as per the defined routing rule.
Thank you for following along with this tutorial. If you have any questions or need further assistance, feel free to reach out. Happy monitoring!