We (Charmed Kubeflow team) are writing sidecar pattern charms for kubernetes native applications and trying to develop nice patterns for this in Juju. The responsibilities of these operators typically include:
- deploying all resources of the application (CRDs, services, …)
- reconfiguring deployed resources on config change (eg: if we start using storage solution B instead of solution A, we need to redefine the deployment)
- enforcing that actual state of these resources matches desired state (eg: services/CRDs exist, deployment actually has N replicas, etc)
- culling resources at application removal
For deploying (1), typically the applications we charm provide their own Kustomize manifests and/or Helm charts that are nearly complete (we might need to change namespaces, configure connections to other applications, add some labels, etc). In podspec patterns we effectively needed to transcribe these manifests into charm.py code so that the charm explicitly created each one, necessitating a lot of error-prone manual work and future charm.py upkeep to follow future versions. Meanwhile, conceptually one could imagine making light modifications to the provided manifests (a few automated search and replaces) then kubectl applying them, which makes you feel there’s an easier way than programming a charm to know how to deploy each piece of the app explicitly. Does anyone have any recommended patterns for generating these sorts of charms?
Similarly for enforcing the actual state (3, eg: reconciliation loops), we could write code that uses the k8s python API to explicitly check for all the necessary resources (Service A, CRD X, …) this app uses to ensure they exist and are up to date, but this again means lots of code maintenance. Meanwhile, conceptually you can again picture doing a “diff” between the manifest you asked for and the cluster state you currently have. It feels here like some tooling can make this pretty easy.
And for updates (2)/removal (4) the situation is also similar - these feel like they could be automated and maintenance burden reduced.
In both cases it feels like there’s a better way, but exactly what that would be is debatable. Ideally we’d be able to rely on the upstream project’s manifests/helm charms and make slight edits as needed. I’d love hearing from others who’ve faced similar situations or anyone who has an opinion. Do you have recommended patterns or tooling for these situations? Thanks!
Andrew