It’s been a while since I looked at k8s charms, but with operator coming along I’m checking it out again. Having written several charms I’ve become accustom to writing a template with decent testing as a first step so that I have a good base to build from. If you’ve seen the current operator template in charm-tools that was one I wrote (and yes it needs updates, I’ll put that on the to-do list).
This weekend I’ve been trying to make a k8’s specific operator template and would appreciate some feedback / review about best practices. I’ll submit it to charm-tools as an actual MR , but wanted to see if there was any input about approach and some of the difficulties I ran across while setting this up.
The template: https://github.com/chris-sanders/template-k8s-operator
Branch on charm-tools with the template available: https://github.com/chris-sanders/charm-tools/tree/k8s
This is the result initialing a charm to see what it looks like: https://github.com/chris-sanders/busybox-operator
If you build the above snap from the charm-tools branch, you can install it and kick the tires. Otherwise take a look at or clone the busybox repo to see what the output looks like.
Folder Structure
I’m using mostly the recommend structure from operator. One thing since I’ve last seen operator is the change to using the “env” folder for dependencies. This template does that with pip3 to install operator. The dependencies are all in a requirements.txt file and the Makefile has a stage to update the folder after updating the requirements.
I ended up leaving the original “lib” folder from the earlier versions of operator. I’m currently using this to ‘vendor’ other code bases that aren’t packaged for install. You’ll see the resource-oci-image repository is setup this way by the template and provides an example how to use git submodule to vendor non-packaged libraries.
I’ve added a Makefile that provides the common tasks like running test, updating the env folder, etc. However, you can run tox directly which is especially useful if you want to pass additional parameters.
Lint
Linting has it’s own tox environment and uses flake8. I’ve basically just used a drop in from previous templates, it’s not passing currently I’ll need to fix some doc strings and fight with import ordering. Anyone particularly familiar or opinionated about making lint and import orders easier to deal with I’m all ears. Otherwise, for the final MR I’ll make a pass and clean up what it’s found.
Unit test
Unit testing is included in the test/unit folder. It’s using Harness and also has it’s own tox environment. Unit tests have their own requirements file to add test only requirements to it’s tox environment. Coverage reporting is also included and an htmlcoverage is generated to view code coverage.
To get harness to work with the oci-image and k8s I had to fork operator. I’ve added some code to populate the necessary resource. I’m curious if this is the right general approach. I’m not yet concerned with how this branch is implemented. If this seems like a necessary addition to harness I’ll look at the implementation details I don’t suspect it will be hard to get from where it is now to an acceptable implementation on harness.
Functional tests
Functional testing is implemented via zaza. The template will setup a bundle and deploy to a k8s cluster. You will have to be sure your local client has the k8s credentials. When you run juju add-k8s you can select to add just to the controller and libjuju won’t be able to interact with the model if you do that.
I’m aware that the folder structure for functional testing is a bit odd. Unfortunately zaza hard codes paths. The template has everything zaza starting from the directory ‘tests/functional’ all paths beyond that are non-configurable zaza paths. If zaza is the right tool I intend to open a bug asking for zaza to allow some better configuration of it’s directory structure to remove some of the redundency.
I have also discovered that juju is only reporting on the status of the “Operator” pod and not the workload pod(s). This results in zaza passing this initial deploy if the operator installs even if the worker pod fails. The template works, but during development I noticed this with image pull failures which zaza/juju were unaware of. Maybe a base part of a k8s-operator charm needs to include some checking/monitoring of the pod(s) to sync that status back to Juju? Is that already available in some form?
Misc
Getting the right pod_spec was super painful. The oci-image library that I’ve included seems like it’s important enough to be part of Operator. Also for a new user, I’d love to include a link to a definitive resource on pod_spec formats and options. Does that exist? Between the juju.is documents (which are all reactive) and multiple forum posts here I guess you can piece together a pod_spec but it’s really very hard.
Feedback
I’d really like to start writing k8s-operator charms with a solid base and approach. I’d also like to save anyone else trying to test out charms for k8s from having to go through the several days of frustration I just did to get a code base that is setup to test and deploy. Please let me know how this could be improved before I work with charm-tools, zaza, and Operator to land necessary forks to make this work out of the box.
For anyone that like me learns best by writing code, getting setup to test K8s charms with whatever software you’re interested in deploying is the best way to get adoption. Hopefully this helps lower the barrier of entry for people so they can see how useful application modeling is with out so much up front investment to just get started.