This document describes the cryptography used by Charmed Kubeflow (CKF).
Resource checksums
CKF uses pinned revisions of Rocks. Charms are refreshed to a newer revision periodically, but each revision is pointing to a specific Rock hash. This provides reproducible and secure environments.
Every artifact bundled into CKF is verified against their MD5, SHA256 or SHA512 checksum after download.
Sources verification
CKF sources are stored in GitHub repositories for Rocks and Charms.
All CKF artifacts built by Canonical are programmatically published and released using release pipelines implemented with GitHub Actions. Distributions, provided as bundles, are published and versioned in tracks via Charmhub, while Rocks and Charms are published using the workflows defined in their respective repositories.
All repositories in GitHub are set up with branch protection rules, requiring:
- New commits to be merged to main branches via pull request with at least one approval from repository maintainers.
- New commits to be signed, for instance, using GPG keys.
- Developers to sign the Canonical Contributor License Agreement (CLA).
Cryptographic processes
CKF is a Juju bundle consisting of several charms integrated using Juju relations. See Kubeflow bundle for more details.
Below is an overview of all cryptographic processes handled by the charms integrated in the CKF bundle.
The charms within the CKF bundle that are not mentioned in this guide do not involve any cryptography in their logic.
admission-webhook
This charm is responsible for applying Kubeflow PodDefaults to newly created Pods, and thus creates a MutatingWebhookConfiguration object for registering a webhook.
The charm code generates a self-signed X.509 certificate, so that the Kubernetes API server can confirm the workload container identity.
dex-auth
This charm is an operator for Dex, which provides authentication to Charmed Kubeflow.
If the Dex built-in connector is used, then the source of truth resides in the connected system, such as Azure Entra ID, AWS Cognito. Dex also supports credentials login via configuration options in the Charms. In this case, the password is always stored as a hash, generated using bcrypt.hashpwd with a Blowfish cipher.
When using the Dex built-in connector, the user password specified in the static_password field is hashed with the bcrypt library before being stored in the Dex configuration.
Additionally, the stored charm state is salted with bcrypt to prevent reverse engineering.
istio-pilot
CKF uses Istio service mesh to enable end-to-end authentication and access control. The workload container of istio-pilot is responsible for distributing an X.509 certificate using sha25withRSAEncryption to every sidecar container in the following path: /var/run/secrets/istio/root-cerm.pem.
When a workload container, i.e., the client, sends a request to another workload container, i.e., the server:
-
Istio reroutes the outbound traffic to the client’s sidecar.
-
The client’s sidecar starts an mTLS handshake with the server’s sidecar.
-
The two sidecars establish an mTLS connection, and Istio forwards the traffic from the client to the server.
-
The server sidecar authorises the request, and forwards the traffic to the backend service through local TCP connections.
The minimum required version of TLS is TLSv1_2.
See Istio Mutual TLS authentication for more details.
Additionally, the charm uses the cert-handler library to generate an X.509 certificate for the Istio Gateway object.
katib-controller
This charm creates a MutatingWebhookConfiguration object that calls a webhook whenever a new Kubeflow experiment is created or updated.
Similar to other charms that create MutatingWebhookConfiguration objects, katib-controller generates a self-signed X.509 certificate so that the Kubernetes API server can confirm the workload container identity.
kfp-persistence
This charm creates a service account token that is used to verify requests to the Kubeflow Pipelines service.
kserve-controller
This charm creates a MutatingWebhookConfiguration object that calls a webhook whenever a new KServe InferenceService is created or updated.
Similar to other charms that create MutatingWebhookConfiguration objects, this charm generates a self-signed X.509 certificate so that the Kubernetes API server can confirm the workload container identity.
minio
This charm is an operator for MinIO, which provides S3 object storage. It uses a field in the object-storage interface named secret-key. Its value is created from a randomly generated 30-character string.
Additionally, the charm adds a randomly generated salt to its configuration before it is hashed with SHA-256, to prevent reverse engineering the secret-key field.
oidc-gatekeeper
This charm uses the client-name and client-secret configuration options for the OpenID Connect client. Similarly to minio, the value of secret-key is created from a randomly generated 30-character string.
pvc-viewer
This charm creates a MutatingWebhookConfiguration object that calls a webhook whenever a new PVCViewer is created or updated.
Similar to other charms that create MutatingWebhookConfiguration objects, this charm generates a self-signed X.509 certificate so that the Kubernetes API server can confirm the workload container identity.