Modifying Grafana dashboards in air-gapped environments

When a charm is integrated with COS Lite, its own bundled dashboards and alert rules are provisioned automatically. However, unless you own the charm, you can’t really modify those things as you want. To apply changes to a dashboard, you’ll either have to manually create it, or to send it over to Grafana via a git repository through the cos-configuration-k8s charm.

But how do you do that? And what if you’re in an air-gapped environment? Let’s go through the solutions in more detail.

Manually adding dashboards

One way of modifying a provisioned dashboard to suit your needs, is to manually create a new dashboard starting from the previous one. To do so, export the dashboard and then import it to create a duplicate you can edit.
Now you can make all the changes you’d like and simply save the dashboard!

The dashboard will be persisted across upgrades, so you don’t have to worry about it disappearing. However, if you’d like a more stable, version-controlled solution, you’ll need to use cos-configuration-k8s.

Setting up COS Configuration with a local repository

The cos-configuration-k8s can be used with any git repository: regardless of it being hosted on GitHub or being on a local machine, as long as the charm can reach it, things will work.

It only takes three simple steps:

  1. Initialize a local git repository somewhere:
    git init <path/to/repo>
    
  2. Deploy COS Config and point it to that local repository:
    juju deploy cos-configuration-k8s cos-config
    juju relate cos-config grafana
    juju config cos-config git_repo=<machine_address>:<path/to/repo>
    
  3. Clone the local repository and push your dashboards and alert rules to it, as you would normally do:
    git clone <machine_address>:<path/to/repo> # e.g., git clone enterprise.github.internal/user/repo
    cd <repo>
    # Make some awesome dashboards!
    git add grafana_dashboards
    git commit -am "add grafana dashboards"
    git push
    

This approach allows you to have version control on dashboards and alert rules even in air-gapped environments.