If you have an Istio deployment in your cluster and have some charmed workloads that are on the mesh, you can configure Istio to enable it to auto generate traces from traffic between your charmed workloads inside the mesh and export them to a tracing backend.
Deploy Istio
First, we need to deploy istio-k8s in an istio-system model.
This would deploy Istio in its ambient mode.
See more: Istio’s ambient mode
juju add-model istio-system
juju deploy istio-k8s istio --channel edge --trust --config auto-allow-waypoint-policy=false
auto-allow-waypoint-policy=false is needed when deploying Istio to allow waypoints to communicate with their workloads.
Add COS to the mesh
The next step is to add some workloads to the mesh and have some traffic flowing inside your mesh.
For this example, we’ll be deploying cos-lite alongside Tempo HA in an example model mesh and add them to Istio’s mesh.
Deploy cos-lite
juju add-model mesh
juju deploy cos-lite --trust --channel edge
Deploy Tempo HA
Check this guide to deploy Tempo HA.
Then run jhack imatrix fill to cross relate everything in the model.
You can also relate the charms separately using juju integrate.
Deploy istio-beacon-k8s
istio-beacon-k8s charm will add charmed workloads deployed on the same model to the mesh. It would also deploy a waypoint proxy, an ambient-mode Istio component that will be responsible for generating and exporting mesh traces.
Read more: Istio’s waypoint proxies
juju deploy istio-beacon-k8s --channel edge --config model-on-mesh=true
Integrate istio with tracing
In the mesh model,
juju offer tempo:tracing
In the istio-system model,
juju consume admin/mesh.tempo
Integrate charm-tracing and workload-tracing
juju integrate istio-k8s:charm-tracing tempo
juju integrate istio-k8s:workload-tracing tempo
Until https://github.com/canonical/istio-k8s-operator/issues/30 is fixed, you’d need to apply this kubernetes manifest
microk8s.kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: tempo
spec:
hosts:
- tempo-0.tempo-endpoints.mesh.svc.cluster.local
ports:
- number: 4317
name: grpc-otel
protocol: GRPC
resolution: DNS
EOF
Visualize traces in Grafana
Open Grafana web UI form your local web browser and login using the admin password.
Then,
Toggle Menu → Explore → Choose your Tempo datasource
This should result in an output similar to the one below:
As expected in Istio’s ambient mode, mesh traces are generated in the L7 layer by the waypoint proxy (i.e istio-beacon-k8s), so you’d find traces generated under istio-beacon-k8s-mesh-waypoint.mesh service name.
Stitching spans together
By design, Istio will not propagate trace headers, so it would be the workload’s responsibility to propagate the trace headers.
Charmed Istio will generate and export traces using OpenTelemetry protocol and therefore, if a charmed workload on the mesh is already instrumented to propagate OpenTelemetry headers, Tempo would stitch together spans generated from that workload with the one generated by its waypoint proxy along with any other workloads that are instrumented in the same manner as you can see in the screenshot above.