Enhancing Multipass Metrics with Prometheus Node Exporter

Metadata
Key Value
Summary Multipass Metrics with Prometheus Node Exporter.
Categories metrics
Difficulty 1
Author Michael Thamm

Contents:

Introduction

When managing a Multipass virtual machine (VM), the multipass info command and the Multipass GUI client, instances tab provides useful metrics. However, from outside of the VM there is much more we can extract that is not exposed by Multipass itself.

The focus of this tutorial is to use the Prometheus snap on your host to monitor metrics exposed by Prometheus node-exporter running as a service within a Multipass VM.

This tutorial assumes you have the following snaps installed on your host: multipass, prometheus, and jq. This can be achieved with:

sudo snap install multipass prometheus jq

Let’s go and monitor some Multipass VMs!

Deploy and Configure a Multipass VM

Create 2 VMs running Ubuntu 22.04 named multipass-vm-1 and multipass-vm-2 with:

multipass launch 22.04 --name multipass-vm-1
multipass launch 22.04 --name multipass-vm-2

Install the prometheus-node-exporter apt package inside the VMs:

multipass exec multipass-vm-1 -- sudo apt install -y prometheus-node-exporter
multipass exec multipass-vm-2 -- sudo apt install -y prometheus-node-exporter

The prometheus-node-exporter apt package runs the binary and exposes it on port 9100

Once the VMs are created, retrieve their IPv4 addresses:

multipass info multipass-vm-1 --format json | jq -r '.info["multipass-vm-1"].ipv4[0]'
multipass info multipass-vm-2 --format json | jq -r '.info["multipass-vm-2"].ipv4[0]'

Replace <VM1_IP> and <VM2_IP> in the YAML content below with the results from above

sudo tee /var/snap/prometheus/current/prometheus.yml > /dev/null <<EOF
global:
  scrape_interval: 15s # The default is every 1 minute.
  evaluation_interval: 15s # The default is every 1 minute.
scrape_configs:
  - job_name: prometheus
    static_configs:
    - targets: ['localhost:9090']
  - job_name: multipass-vm-1
    static_configs:
    - targets: ['<VM1_IP>:9100']
  - job_name: multipass-vm-2
    static_configs:
    - targets: ['<VM2_IP>:9100']
EOF

To reload the config file, restart the Prometheus snap:

sudo snap restart prometheus

Browse Prometheus Graphs

The following table provides two sample metric expressions with links that can be opened in a browser. The browser accesses the Prometheus service running on http://localhost:9090. Try them to see the results!

Metric Meaning
rate(node_cpu_seconds_total{mode="system"}[1m]) The average amount of CPU time spent in system mode, per second, over the last minute (in seconds)
rate(node_network_receive_bytes_total[1m]) The average network traffic received, per second, over the last minute (in bytes)

Select the Graph tab in the web UI to see the results:

Now you have real-time metrics, scraped and evaluated, every 15 seconds by Prometheus in a nice web UI that leverages the node-exporter capabilities, have fun observing!

Grafana Agent Forwarding to Prometheus

The previous setup is fine, but we can improve our observability by introducing Grafana Agent to our architecture. Grafana Agent is a telemetry collector for sending metrics, logs, and trace data to:

  • Prometheus ecosystem
  • OpenTelemetry ecosystem
  • Grafana ecosystem (Loki, Grafana, Tempo, Mimir, Pyroscope)

For the scope of the tutorial, we will send metrics to our Prometheus snap from before.

First, install the Grafana Agent snap with:

sudo snap install grafana-agent

Then, update the config file to scrape the metrics of the VMs and Grafana Agent itself. Remember to replace <VM1_IP> and <VM2_IP> in the YAML content:

sudo tee /var/snap/grafana-agent/current/etc/grafana-agent.yaml > /dev/null <<EOF
metrics:
  global:
    scrape_interval: 15s
  configs:
    - name: agent
      scrape_configs:
        - job_name: grafana-agent
          static_configs:
            - targets: ['localhost:12345']
        - job_name: multipass-vm-1
          static_configs:
            - targets: ['<VM1_IP>:9100']
        - job_name: multipass-vm-2
          static_configs:
            - targets: ['<VM2_IP>:9100']
      remote_write:
        - url: http://localhost:9090/api/v1/write
EOF

Since we are moving the scraping responsibility to Grafana Agent, the Prometheus config file can be updated by removing the scrape_configs for the Multipass VMs:

sudo tee /var/snap/prometheus/current/prometheus.yml > /dev/null <<EOF
global:
  scrape_interval: 15s # The default is every 1 minute.
  evaluation_interval: 15s # The default is every 1 minute.
scrape_configs:
  - job_name: prometheus
    static_configs:
    - targets: ['localhost:9090']
EOF

To enable remote-write for Prometheus, we need to modify the daemon_arguments file with:

sudo sed -i 's/^ARGS=""/ARGS="--web.enable-remote-write-receiver"/' /var/snap/prometheus/current/daemon_arguments

To apply the configuration changes, restart the Prometheus snap:

sudo snap restart prometheus

Warning: The file daemon_arguments file states:

Due to snap confinement the following options should not be set

For this tutorial, the warnings can be ignored since the /api/v1/write endpoint is on the same host port 9090 and we are not exposed to the outside world. In a production scenario TLS, authentication, and access control should be considered.

With the config files set, we can start grafana-agent:

sudo snap start grafana-agent

Finally, check that Grafana Agent is correctly writing to Prometheus with this link:

rate(node_cpu_seconds_total{mode="system"}[1m])

As you can see, there was a section where metrics were not being received by Prometheus. This section is highlighted in purple and is the time it took to switch over from Prometheus to Grafana Agent for scraping.

Note: We can check which jobs are being scraped with http://localhost:9090/api/v1/label/job/values

Next steps

Host Environment Cleanup

To return your host environment back to the original state, remove the Multipass VMs:

multipass stop multipass-vm-1 & multipass delete -p multipass-vm-1
multipass stop multipass-vm-2 & multipass delete -p multipass-vm-2

Remove the snaps installed for this tutorial,

Warning: Run multipass list before continuing to ensure that you do not have any important VMs running in Multipass before removing the snap!

sudo snap remove --purge multipass prometheus grafana-agent jq
4 Likes

A copy of this tutorial also lives in the Multipass Discourse now, so that our users can find it easily if needed.

1 Like