Hi,
In the recent and very useful observability workshop, I raised a question around Loki and some problems I’ve faced with COS-lite and it was suggested to make a discourse post. @erik-lonroth @0x12b @jose @ppasotti
The context is that after instrumenting my machine charm with grafana-agent and setting up a local COS-lite environment, I was able to see logs in Grafana and proceeded to setup a few panels that displayed logs for different sources - syslog, model logs, unit logs, etc. Mainly to observe what the difference was between them. That’s when I ran into a bit of an issue, my Grafana panels all timed out and displayed an error “too many outstanding requests”.
Further investigation landed me on a few threads - like - this suggesting changes to Loki config.
The Loki charm currently doesn’t offer much in the way of config so my question is how will users of the COS-lite stack be able to tune their deployment. My understanding is that sensible charm defaults should get one most of the way there but predictably there might be a need to configure parameters for large and small scale operations.
Thanks for the amazing work on COS-lite so far!