The Future of Charmed Operators on Kubernetes
With the release of Juju v2.9 RC7, we’re previewing the future of Charmed Operators on Kubernetes by introducing sidecar charms that are more consistent with how workloads are managed across other Juju substrates.
Rationale and History
With the first generation of charms on K8s, Charmed Operators ran in their own Pods, and instructed Juju to provision the workload on the Kubernetes substrate through a mechanism called
pod.set_spec. Workloads were then provisioned into their own Pods, separate from the Charm code. This approach had some inherent limitations:
- Limited control over processes in the workload
- No IPC or local communication between the Charmed Operator and the workload
- No file or socket sharing
- Inability to store per-unit state
Charmed Operators implementing this pattern are susceptible to more of the challenges associated with distributed computing, especially in those cases where Charmed Operators were not running on the same Kubernetes nodes as any or all of the workload Pods (due to the design of the Kubernetes scheduler).
Charms written in this way will continue to work going forward, but we strongly encourage developers to adopt the new sidecar-based approach, and help us make it the best way to operate workloads on Kubernetes or otherwise.
A Refined Approach
With the new approach, both the workload container and the charm run in the same Pod, implementing the Sidecar Pattern. By definition, the Sidecar Pattern is designed to allow the augmentation of workloads with additional capabilities and features - in this case the ability to effectively manage and operate complex workloads and yielding a number of advantages:
- Charmed Operator and workload will always be scheduled on the same node
- Charmed Operator and workload are co-located in same network namespace
- Charmed Operator and workload can communicate with SHM or sockets
- Files can be shared between Charmed Operator and workload more easily
- Charmed Operator scales with the workload
To augment this approach we’ve developed Pebble: a lightweight, API-driven process supervisor designed for use with modern Charmed Operators.
Pebble enables you to declaratively configure processes to run in a container, and control those processes throughout the workload lifecycle. It features a layering system that allows for coarse revisions to running configurations.
When writing a Charmed Operator that implements the Sidecar Pattern, no modifications are required to the base container images.
How It Works
Juju automatically injects Pebble into workload containers using an initContainer and Volume Mount. The entrypoint of the container is overridden so that Pebble occupies PID 1. Pebble is controlled by the Charmed Operator using a UNIX socket, which is mounted into both the Charmed Operator container, and the workload container. The Charmed Operator communicates over the socket with Pebble to manage running workloads.
Example Sidecar Charms
To help you get started implementing Charmed Operators using this new approach, you can follow the conversion process of some existing charms. This list will be kept up to date as we progress.
- benhoyt/snappass-test - Concept Demonstration
- mthaddon/charm-k8s-gunicorn - WIP
- jnsgruk/charm-jnsgruk-k8s - WIP
- martinrusev/grafana-operator - WIP
You can use any existing bootstrapped Kubernetes cluster, provided the controller is at least version 2.9. See the documentation for instructions on how to upgrade your controller. If you do not have a cluster ready, you can use MicroK8s!
$ sudo snap install --classic microk8s $ sudo usermod -aG microk8s $(whoami) $ sudo microk8s enable storage dns $ sudo snap alias microk8s.kubectl kubectl $ newgrp microk8s
Once you’ve done that, you should be able to invoke
microk8s commands without using
sudo. If you can’t, try logging out and logging back in before continuing.
Next, let’s install and bootstrap Juju, then deploy an example Charmed Operator:
# Make sure we have the correct version of Juju installed $ sudo snap install juju --classic --channel=2.9/edge # Bootstrap a Juju controller on MicroK8s $ juju bootstrap microk8s # Install Charmcraft $ sudo snap install charmcraft --edge # Clone an example charm $ git clone https://github.com/benhoyt/snappass-test # Build the charm $ cd snappass-test $ charmcraft build # Create a model for our deployment $ juju add-model snappass # Deploy! $ juju deploy ./snappass-test.charm \ --resource snappass-image=benhoyt/snappass-test \ --resource redis-image=redis # Wait for the deployment to complete $ watch -n1 --color "juju status --color"
You can now inspect your deployment with kubectl:
# List pods in the snappass namespace $ kubectl -n snappass get pods NAME READY STATUS RESTARTS AGE modeloperator-5cd65496c-6t6sq 1/1 Running 0 3m snappass-test-0 3/3 Running 0 1m
Note that the
snappass-test-0 pod indicates 3 running containers, in this case these are:
- Charm container
- Snappass container
- Redis container
You can see the Pebble configuration for the snappass and redis containers in the Charm code. The Charm container was injected automatically by Juju.
Where to get help
We’re hard at work augmenting the existing Operator Framework and Juju documentation to include these new APIs. If you need help with your new Charm, write a post on the Charmhub Discourse, or reach out on the Charmhub Community Mattermost instance.