Secrets and Volume Mounts for K8S Charms

charm
,
1

I have been poking around trying to see what it would take to get the elastic-operator deployed via Juju. I think I’m hitting a few limitations of k8s charms. Hoping I can shed some light on what I’m seeing and get some feedback.

The elastic-operator charm I’m working on can be found here. The purpose of this charm is to be able to juju deploy the elastic-operator here. Outside of having to kubectl apply the juju-unsupported objects by hand prior to deploying the charm, things are coming along pretty smooth. I am currently stuck trying to get the /tmp/cert secret mount configured and populated in the operator, see traceback pastebin. Not really sure how this would be accomplished via Juju, so I started digging. My primary goal is to understand how I might create a secret and mount it as a volume in a pod via configuration via Juju.

I thought it would be helpful to see what a pod storage configuration of a juju deployed k8s charm looked like from k8s side, so I deployed mariadb-k8s with juju deploy cs:~juju/mariadb-k8s-2.

Running microk8s.kubectl edit statefulset.apps -n mymodel mariadb-k8s to view the deployment configuration shows this.

The first thing that stuck out to me was that the name of the volume mount is comprised of the charm storage name concatenated with a uuid, ex database-45300d63.

This makes me believe that this is the portion of the elastic-operator yaml that I will need to figure out how to fit into the way juju creates a k8s deployment. I’m not sure how to go about referencing the volume name in the pod spec because it seems generated with a uuid that I don’t when creating the pod spec.

For reference, this is the elastic-operator deployment created from the command kubectl apply -f https://download.elastic.co/downloads/eck/0.9.0/all-in-one.yaml.

From my user perspective, I feel that juju/k8s-charms need some bindings to facilitate secrets and secret volume mapping. Thoughts? Ideas?

The other pitfalls I’m experiencing seem to fit into the same category; k8s objects that I don’t know how to model with a charm.

An example of this is a serviceAccount. How can I create a serviceAccount from a k8s charm?

This same issue persists for me with all other objects that I am trying to create; secrets, serviceAccounts, clusterRoles, clusterRoleBindings.

What is our approach to handling the creation and mapping of these objects from Juju’s perspective?

Thanks

2

I have this dialed in a little bit more now.

Maintaining the Upstream YAML

I thought long and hard about how to get the upstream elastic k8s yaml files into this charm without having to maintain them in the charm. I think it makes sense to pull them in from the upstream elastic repo at charm build time and add them to the files dir of the charm. The charm code could then reference the needed yaml files when assembling the pod spec. I hadn’t written a tactic before, but with some careful introspection of @cory_fu’s handy work on the wheelhouse tactic, I started to see how a tactic might come together for my use case with the k8s yaml. The tactic I made for the elastic-operator charm can be found here.

Assembling the Pod Spec

Following @knkski’s lead I made a function that assembles and returns the pod spec as a python dict that reads in the crd resource files and operator template file brought in by the tactic. It then parts out and adds the needed bits to the pod spec. My make_pod_spec() and yaml converting helpers here.

I feel pretty solid about where this is going, I’m just a little unsure how to assemble what seem to be the final bits needed to get this operator deployed and running via Juju.

I know secrets are a new and moving target, so I wanted to hopefully get some feed back from others concerning how we might facilitate secrets mapping to volumes from a user’s perspective.

The use case driving this is the one described above, where my operator charm needs to be able to assemble and apply the bits here.

I feel like I’m getting close to having a charmed elastic-operator.

Thoughts/ideas on how we might be able to facilitate these bits?

Thanks

3

Hi @jamesbeedy,

Do the new changes in the v2 podspec help? It now has secrets.

4

@thumper I can create secrets via my charm’s pod spec - which is awesome, and definitely one step in the right direction :clap::clap::clap::clap::clap::raised_hands::raised_hands::raised_hands::facepunch::facepunch::facepunch::facepunch:

What I don’t understand how to do via a k8s charm is map a secret to a volume

Upstream Docs: Using Secrets as Files from a Pod

5

Juju doesn’t currently model secrets.

If secrets are considered as a source of read only data, one option might be to use a storage abstraction to mount that data into the pod. This is similar to providing read only training data to an AI model.

Juju currently sets up volumes and volume claims for a pod based on Juju’s storage model, which allows for 3 types of storage provider:

  1. “kubernetes” which maps to a k8s storage provisioner (via an optional Juju storage pool)
  2. “tmpfs” which maps to k8s emptyDir in memory
  3. “rootfs” which maps to k8s emptyDir on disk

The “kubernetes” storage provider for Juju effectively sits on top of a Juju storage pool. We could look to enhance the storage-provisioner attribute of a storage pool to allow a pool intended for use with k8s to add support for referring to a secret rather than a k8s storage provisioner or storage class.

Note:
The above though really only solves the problem based on primitives available in k8s, rather than a more holistic “how should Juju model secrets” approach. And it is indeed clunky as it requires work by the user outside the charm where the secret is defined in order to wire things up (this is really a show stopper for this approach).

So although a “workable” stop gap, we should look to how secrets would be modelled and work towards that. This doesn’t solve your immediate problem though. And herein lies the problem. How to find the balance between supporting existing upstream images so stuff works today, whilst encouraging the adoption of more Juju-centric approaches, and filling in the Juju model gaps.

6

100%

Well put - agreed.

All that said, possibly we should put something in place similar to what you are talking about to support the use case. Even if the solution is clunky and doesn’t fit how we want to ultimately model secrets via Juju, it will allow for us to move forward in supporting the elastic stack in the mean time. We could then refine the implementation, user story and work out the kinks to model secrets via Juju appropriately.

7

@wallyworld This use case just disappeared from the upstream operator. :slight_smile:

8

Good :slight_smile: Means we don’t have to try and find a work around and have the time to model things properly.

9

Did you ever find a solution to this ? I am also in need of modeling serviceAccounds, clusterRoles and clusterRoleBindings in a charm and I find no documentation supporting this.

10

Your charm can deploy secrets, service accounts, and cluster roles even though those are not yet modelled by Juju. You inlude those in the optional k8s specific YAML file passed to k8s-spec-set.
There’s more below. If there’s a particular use case where more info is needed, maybe provide an example of what’s needed and we can help.

and here

and here