[Tutorial] Using Juju on Charmed Kubernetes on VMWare

This is in connection to the topic of Using Kubernetes with Juju. See that page for background information.

This tutorial explains how to deploy a Kubernetes cluster on VMWare vSphere and configure Juju so that it can deploy k8s charms on that cluster.

If you have any comments or suggestions, let me know!

Requirements

You need a working Juju controller connected to a working VMWare vsphere cluster, and have switched to the model you want to deploy the Kubernetes cluster to.

More information on setting up a Juju controller on VMWare vsphere. If you can’t get Juju to work on VMWare vsphere, let me know and I’ll add the bootstrap instructions we use.

1. Deploy Kubernetes on VMWare

I’m using the Kubernetes Core bundle, but this should also work with the Charmed Distribution of Kubernetes.

The defaults of these bundles are good, so you can just deploy them.

juju deploy cs:bundle/kubernetes-core

2. Deploy Ceph

We’ll be using Ceph for persistent storage. Persistent storage is required for running Juju on Kubernetes.

In this example we’ll be installing Ceph on VMWare too. This setup isn’t optimal however, because this creates an additional layer of indirection in storage. It’s better to use an existing Ceph cluster deployed on bare metal.

# Deploy the ceph cluster
juju deploy -n 3 ceph-mon --constraints "mem=1G"
juju deploy -n 3 ceph-osd --storage osd-devices=200G,2 --storage osd-journals=8G,1 --constraints "root-disk=500G mem=1G"
juju add-relation ceph-osd ceph-mon

# Connect the Ceph cluster to Kubernetes, this will add the Ceph cluster as "Storage Classes" to k8s
juju add-relation ceph-mon:admin kubernetes-master
juju add-relation ceph-mon:client kubernetes-master

These commands add ceph as the default storage class to k8s. Juju will use the default storage class in k8s if no other storage pool is configured.

More info: attaching Ceph storage to Charmed Kubernetes
The constraints are derived from the hardware recommendations for Ceph clusters.

3. Wait

Wait until the entire deployment is complete before proceeding to the next step. The following command shows you the status and updates every 2 seconds.

watch -c juju status --color

4. Download the Kubernetes kubectl config

You need the right info to connect Juju to the k8s cluster.

juju scp kubernetes-master/0:config ~/.kube/config

You can find more info on getting started with kubectl and the Charmed Kubernetes bundles in the READMEs of those bundles.

5. Add the K8s cluster to the existing Juju controller

Now you can add the k8s cluster as a “cloud” to the controller. This will allow you to create models juju models on that cluster.

juju add-k8s k8s-test-cloud

This command uses config you just downloaded to ~/.kube/config.

6. Create a k8s model on the Juju controller

You can create a k8s model on the controller by specifying the k8s cloud you just added.

juju add-model k8s-test k8s-test-cloud

7. Deploy an k8s charm in the Juju model

At this point you should have a working model, ready to deploy charms. Test it out by deploying MariaDB.

juju deploy cs:~juju/mariadb-k8s mdb3

Note that we don’t specify which storage the database should use. This will cause Juju to use the default k8s storage.

8 Likes

This is super interesting. I know of a college at work who I think will love this and maybe give it a go.

Hi Merlijn,

Thanks for the tutorial, but I wanted to point out that it is probably a better idea to use the vSphere integrator charm (vsphere integrator | Juju) when running Charmed Kubernetes on-top of VMware/vSphere.

The main reason for this is that Ceph is a software-defined storage mechanism with replication and that usually the underlying storage for your vSphere is also replicated and thus, you have two layers of replication for the same data which is not very optimal.

The VMware integrator charm allows you to integrate Charmed Kubernetes directly to vSphere using the Kubernetes cloud provider for VMware which enables the ability to utilise VSAN or other storage provided directly by the VMware environment.

The main caveat however is the fact you need a privileged account which is able to provision storage (as described here: https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/vcp-roles.html).

I would say the main reasons for running Ceph on-top of VMware is to either:

  1. Duplicate a production environment which runs on bare-metal for test or dev
  2. Environment where admin account with permissions are not available to provision storage on vSphere
  3. You need some kind of storage/API which cannot be provided by underlying storage like NFS or S3 protocol

Cheers & Thanks for the tutorial,

  • Calvin
2 Likes

Thanks, calvin!

This tutorial was written before the integrator charm existed. Feel free to update the post, it should be editable by everyone.

2 Likes