K8S Charm Questions - Jupyter Enterprise Gateway Operator

framework
, ,
1

Intro

My goal is to understand how to use the new pebble functionality to create a k8s operator charm. I am targeting the jupyter-enterprise-gateway workload for my example, and am hoping to get some feedback in areas where I don’t fully understand how to proceed.

Find the k8s yaml for the jupyter-enterprise-gateway here

Purpose of the charmed operator

In the context of jupyter-enterprise-gateway, I see the charm serving two primary functions:

  1. provision the resources in k8s that are required by for the workload to run
  2. facilitate operation of the workload inside of k8s

Resources

The resources that jupyter-enterprise-gateway depend on (from the k8s yaml above):

Picking these k8s resources apart one by one and mapping each of the them to their counterpart in Juju and the Operator Framework will give us a mapping of how the resources can be assembled.

Namespace

The Namespace is created by juju when you create a model on the k8s cloud. The name of the Namespace can be accessed via the environment variable JUJU_MODEL_NAME or through ops using self.model.name.

ServiceAccount

ClusterRole

ClusterRoleBinding

Service

Deployment

DaemonSet

Fallout

Namespace is the only k8s resource I can identify right now. I don’t know where to look to get information about how to proceed with assembling the others using the Operator Framework. I see set_pot_spec(), but am unsure how to use this with pebble. Searching for ServiceAccount in the sdk docs doesn’t give much.This highlights a place where documentation could be helpful, “Mapping k8s resources to charmed operator primitives”.

I would like some feedback about how to proceed with provisioning the resources needed to run the jupyter-enterprise-gateway workload using the Operator Framework.

Operation of the k8s workload

Operating the jupyter-enterprise-gateway workload inside of k8s would consist of two responsibilities as far as I see it:

  1. Lifecycle operations for the containers in the Deployment (the containers that supply the enterprise gateway api endpoint), and
  2. Lifecycle operations for the kernel-image-puller DaemonSet.

This is another juncture where I’m not sure what to do and thus anther opportunity to enhance the documentation, “DaemonSets and Deployments using the Operator Framework” would do a lot of good here.

Conclusion

Given this short experiment, I am looking for feedback on two things:

  1. How do I provision k8s resources required by the workload via primitives in the Operator Framework?
  2. How do I use pebble to manage the workload; Deployment, DaemonSet or both?

Thank you!

(Edit) After giving some more thought, I’m inclined to think that the kernel-image-puller DaemonSet and the enterprise-gateway-api Deployment might be best modeled as two separate charms.

2

Hi James,

I think your thought about separating this into two charms makes sense. However, see the note below about Deployments and DaemonSets.

To go through the k8s resources that you’re looking at based on the upstream yaml file:

  • Service Account
    A service account will be created for you - if it needs elevated privileges you can ask the operator to run juju trust ${application} --scope=cluster as we do in the nginx-ingress-integator charm.

  • ClusterRoles and ClusterRoleBindings
    Juju will provision these for you. Depending on exactly what you need you may need to talk to the k8s API to modify this. There was an issue where this wasn’t being created, but this seems to have been recently fixed per Bug #1934180 “Juju 2.9 failing to create ClusterRoleBinding” : Bugs : juju so if you’re using 2.9.8 or later you should be fine.

  • Service
    Juju will create a kubernetes service for your charm. Currently it’s not configuring it with the correct ports per Bug #1920960 “ports not tracked with sidecar charms” : Bugs : juju but if you add a relation to the nginx-ingress-integrator charm it will create one for you, or you can do it yourself by connecting to the kubernetes API.

  • Deployments and DaemonSets
    These aren’t currently supported in sidecar charms - all pods are created as StatefulSets currently. I understand there are plans to support DaemonSets, but my understanding of Deployments is that’s still under discussion as to whether it would be supported.

Also, just to clarify one other thing, pod.set_spec is no longer possible in sidecar charms. At the point where your charm code is running the pod has already been created. This is a deliberate design change from the original way Kubernetes charms worked. Previously your code ran in an “operator” pod, and ran pod.set_spec to deploy your workload into kubernetes. With sidecar charms Juju deploys pods which contain the charm code and your charm’s workload as containers that are sidecars to each other.

Hope that helps?

1 Like