Duplicate values working with COS and Grafana dashboards

getting started
#1

I’m trying to understand how a good working process looks like for integrating with grafana-agent juju libs and COS. Its getting promising and I’ll write up my experiences down the road.

I have an example charm which I’ve got to work largely, but I’m stuck at the moment with how I should work with grafana dashboards.

My initial question (I have a few more) is why I seem to get duplicated values for my single-value metrics from the prometheus datasource coming from the COS integrations.

Below is my situation where there are two series where I would like only one:

image
image848×361 26.8 KB

There are two identical series shown and I can’t understand why.

This doesn’t show up when I “Explore” my data as shown below, but turns up like the above image when “Add to new dashboard”

image
image1641×1050 95.2 KB

So, is there anyone that can tell me whats going on from the experts in the community?

#2

Hi Erik,

Would you mind showing the configuration of the panel where you get the duplicates? Ie. the PromQL query and any other configuration you’ve made of the panel.

Best, Simon

1 Like
#3

I have a charm which contains all of it actually which might help.

https://github.com/erik78se/juju-operators-examples/tree/44c2cfa208473b8d9d620f86bdfbb0602e856049/observed

#4

This looks reasonable to me. Unless the query returns two time series it shouldn’t create two lines in the graph. Mind copying the full series name from the legend? What I’m mainly interested in is if any of the labels differ between the green and yellow one.

#5

image
image1660×396 59.1 KB

I can’t see any differences between the two labels. @0x12b

#6

I’ve deployed the same setup on my local lab and played around a bit.

The reason why you’re seeing two time series is because you have the query type set to Both, meaning it will do both a range query and an instant query. If you only want one time series, pick the one that you think makes the most sense for your metric.

As a side note, the metric you’ve picked is a counter (as indicated by the _total suffix), meaning that unless you want it to just be a monotonically increasing graph, you’d need to apply some function like rate or difference to it so that you instead get some kind of delta value. Likely something along the lines of what is attached below:

20230529_10h51m58s_grim
20230529_10h51m58s_grim1051×909 66 KB

3 Likes
#7

I couldn’t see an explicit Both in the dashboard.

Not being a Grafana expert I had a poke around to understand it for my own benefit:

You appear to use only range on your main branch.

2 Likes