In this tutorial you will deploy Parca, a continuous profiling solution by Polar Signals and use it to gather profiles from Grafana, a component in the Canonical Observability Stack.
This tutorial assumes you already know how to write charms using Juju. If you need to first learn how to write a charm, start with From zero to hero: Write your first Kubernetes charm. The goal is to enable the native instrumentation capabilities of an application (via charm code) and ensure the charm can integrate with Parca over parca_scrape.
Read more: What is profiling? (parca.dev)
Prepare the environment
You will need:
- A Kubernetes cluster with a bootstrapped Juju controller and a model.
- Familiarity with Juju
- Charmcraft installed in your environment.
- Some previous charming knowledge
Deploy COS Lite
In your juju model, run the following command:
juju deploy cos-lite --trust
Wait until the model settles. Then, run
juju run grafana/admin get-admin-password
To get the URL where Grafana is deployed and access its UI. Use admin as your username and the provided password to log into grafana.
Deploy and integrate Parca
Run the following command to deploy Parca:
juju deploy parca-k8s --channel edge --base ubuntu@24.04 && juju integrate parca-k8s:grafana-source grafana
Navigate to Grafana and verify that there is a Parca data source in the Explore tab.
Fetch the charm
Clone the grafana-k8s-operator repository in a version that didn’t contain profiling instrumentation yet:
git clone git@github.com:canonical/grafana-k8s-operator.git && git checkout rev129
Take a moment to get acquainted with the repository structure. This repository follows the usual charming practices and contains everything that’s needed to charm Grafana, an open source observability platform.
Introduce profiling
In the next steps, you will use the existing profiling feature in Grafana to enable a profiling endpoint that can be scraped by Parca.
Fetch libraries
In the charm repository, run the following:
$ charmcraft fetch-lib charms.parca_k8s.v0.parca_scrape
You should see something like:
Library charms.parca_k8s.v0.parca_scrape version 0.x downloaded.
Your lib/charms should now have a new parca_k8s library.
Add the endpoint
In charmcraft.yaml, in the provides section, add the following:
profiling-endpoint:
interface: parca_scrape
description: |
Links to grafana's pprof endpoint. Can be used to integrate with parca to
profile the grafana server process.
Update charm code
In src/charm.py, do the following modifications:
- import the newly fetched library into the file. In the imports section, add:
from charms.parca_k8s.v0.parca_scrape import ProfilingEndpointProvider
In the charm.GrafanaCharm.__init__ method, add instantiating ProfilingEndpointProvider. It can be added right after the tracing objects:
[...]
self.workload_tracing = TracingEndpointRequirer(
self, relation_name="workload-tracing", protocols=["otlp_grpc"]
)
self.charm_tracing_endpoint, self.server_cert = charm_tracing_config(
self.charm_tracing, CA_CERT_PATH
)
# new section - profiling
self.profiling = ProfilingEndpointProvider(self, jobs=self._profiling_scrape_jobs)
In the observations section of the charm.GrafanaCharm.__init__ method, register event handlers responding to the profiling endpoint created|broken events so that the profiling configuration is added when the charm is related to Parca and removed when the relation is broken:
# -- profiling observations
self.framework.observe(
self.on.profiling_endpoint_relation_created, # type: ignore
self._on_profiling_endpoint_created,
)
self.framework.observe(
self.on.profiling_endpoint_relation_broken, # type: ignore
self._on_profiling_endpoint_broken,
)
In the main charm class, between the other charm.GrafanaCharm._on... functions, add the profiling event handlers:
def _on_profiling_endpoint_created(self, _) -> None:
"""Turn on grafana server profiling."""
self._configure()
def _on_profiling_endpoint_broken(self, _) -> None:
"""Turn off grafana server profiling (if no other profiling relations exist)."""
self._configure()
In this tutorial, profile endpoint is exposed on port 8080. This is configurable in most applications including Grafana.
Add also the charm.GrafanaCharm._profiling_scrape_jobs referred to in ProfilingEndpointProvider declaration so that Parca knows where to fetch the profiles:
@property
def _profiling_scrape_jobs(self) -> list:
job = {"static_configs": [{"targets": ["*:8080"]}], "scheme": self._scheme}
return [job]
You also need to configure Grafana so that it exposes the profiling endpoint. This is often done using environment variables. In the charm.GrafanaCharm._build_layer method, right before declaring the Layer object, you will add the required environment variables if there is a relation to parca:
# if we have any profiling relations, switch on profiling
if self.model.relations.get("profiling-endpoint"):
# https://grafana.com/docs/grafana/v9.5/setup-grafana/configure-grafana/configure-tracing/#turn-on-profiling
extra_info.update(
{
"GF_DIAGNOSTICS_PROFILING_ENABLED": "true",
"GF_DIAGNOSTICS_PROFILING_ADDR": "0.0.0.0",
"GF_DIAGNOSTICS_PROFILING_PORT": "8080",
}
)
Confused or lost in the implementation? See the pull request in grafana-k8s-operator repository that introduced profiling support. This tutorial mimics the changes from this pull request by doing them step-by-step.
See it in practice
Pack the charm
In the charm repository, run
charmcraft pack
and wait for it to finish.
Refresh Grafana and relate to parca
Run the following command:
juju refresh grafana --path ./grafana-k8s_ubuntu-20.04-amd64.charm
Wait until the charm gets into active/idle state using juju status --watch 2s. Then, relate grafana to parca using the new profiling-endpoint:
juju relate grafana:profiling-endpoint parca-k8s
See the profiles
Navigate to Grafana → Explore tab. Choose Parca as a data source and process_cpu -> cpu in the configuration, then click on “Run query”. You should see the profiling CPU data:
Congratulations, you added a profiling integration to a charm!
Next steps
Take a look at the introduced profiles. Do you see any patterns / potential issues that can be fixed in the upstream application?
Consider adding profiling to your charm as a next step.
My workload doesn’t support profiling
There’s two main paths you can take from here. You can either add profiling support to your workload (manual instrumentation) or use the Parca Agent charm to get profiles from the host your workload runs on.