cos-proxy-operator - docs - Troubleshooting

charm
, ,
1

cos-proxy services are running

Confirm the nrpe-exporter is functioning:

juju ssh cos-proxy-monitors/0 systemctl status nrpe-exporter

Confirm vector behaves as expected:

systemctl status vector
journalctl -u vector
vector top

In vector top you should see reasonable incoming and outgoing traffic.

Try to run NRPE checks manually

Confirm the checks run and produce expected output, for example:

$ /usr/local/lib/nagios/plugins/check_systemd.py aodh-listener
OK: aodh-listener.service is running

cos-proxy has COS relations

Inspect juju status to make sure that cos-proxy has cross-model relations to COS charms:

Integration provider                    Requirer               Interface          Type         Message
cos-proxy:downstream-prometheus-scrape  prom:metrics-endpoint  prometheus_scrape  regular

Similar for loki, grafana.

If cos-proxy is related to scrape-config, then make sure scrape-config is related to prometheus.

Rule files and scrape configs reached prometheus

Make sure the the expected rule files appear are forwarded to prometheus over relation data:

juju show-unit prom/0 --format=json \
  | jq '."prom/0"."relation-info" | .[] | select(.endpoint == "metrics-endpoint")."application-data"'

and that the same rules were written to disk:

juju ssh --container prometheus prom/0 ls /etc/prometheus/rules

and that the same rules were picked up by prometheus:

curl $(juju status --format=json | jq -r '.applications.prom.address'):9090/api/v1/rules | jq

Are nrpe metrics being scraped by prometheus?

  • In the cos-proxy VM, run vector top and look at the “Bytes” column for prometheus exporter. If it isn’t growing then metrics are not being scraped by prometheus.
  • In prometheus, confirm that there are targets (scrape jobs) for the nrpe metrics:
    • In prometheus web UI, under Status → Targets; or
    • curl :9090/api/v1/targets
  • Can the scrape targets be reached from within prometheus? (A manual curl would do.)
  • Check if prom has any command_* metrics.