I’m looking for general advice and best practice information for Proxmox-MaaS-Juju.
I have a three node Proxmox cluster in a VRTX with 24 CPU, 512Gb RAM, and 5Tb shared Ceph. It might not be an ideal configuration, but it’s been pretty stable for about 8mo and Proxmox has earned some credibility in my book.
It’s kind of a sandbox for testing migrating workloads to kubernetes. Very early stages so I’ve tried a number of options for deploying\managing clusters. MaaS-Juju just seems really intuitive to me, at least conceptually, so I’ve been using it to spin up clusters for further testing. It’s pretty nice to be able to spin-up\tear-down multi-node clusters with a few commands.
I have one pve-vm running the MaaS controllers, and about two dozen pve-vms commissioned into MaaS. I’m able to add a MaaS cloud, bootstrap a juju-controller, and deploy charms fairly consistently. Even as-is it’s a pretty useful sandbox, but I think I have a lot to do to make this anything other than a sandbox.
The main hurdle seems to be Proxmox integration. There doesn’t seem to be a comparable integrator charm for Proxmox so I’m struggling with provisioning storage and understanding how I might dynamically provision new vms. I’ve even wondered if maybe I should be trying to use the LXD integrator to work with Proxmox LXC’s, but I’m more familiar with VM’s so I started there.
So this is kind of a general request for advice or direction. Even open to suggestions that this is just a bad combination and I should look for a different stack. But I do like Proxmox and I see some potential there even without MaaS-Juju, and vice-versa, so I’m hoping they can be friends.
Also I’ll toss out that my ultimate goal is to use kubevirt to run Windows workloads in Linux containers to get away from mixed-node clusters, but that’s way, way off so this is more about the Proxmox-MaaS-Juju part.
I do really love ProxMox too, this is a strong and efficient platform, now that I’m working on Openstack, I admit that I miss it very much even though this is far from providing the same level of functionnalities.
Unfortunately, as good as it is, ProxMox is not a first class citizen in the Virtualization world (which is unfair) and it MaaS don’t propose ProxMox as a provider for VM provisionning as it does with Libvirt and LXD.
I suppose Canonical developers just don’t want to support every hypervisors out there otherwise, anyone might ask for VMWare, Hyper-V and many more but I suppose that if there are any interest and some qualified developers willing to take that in charge, I doubt they will refuse any contribution towards that.
Figure I’ll just use this thread to post general notes about this software collection.
The biggest problem I’m having moving forward with PVE as the metal layer is k8s storage provisioning. I’ve tried several promising options, but haven’t been able to get anything fully functional. I’d prefer an ‘external’ storage solution, but I’ll take anything that can successfully provision volumes for now.
STORAGE:
~Ceph-proxy - (ceph proxy | Juju) seemed my best bet and I was able to configure the fsid\key\host from my PVE Ceph config, deploy the charm, and add the relation to kubernetes, but I must be missing a step because it hasn’t created any storage classes and my k8s troubleshooting skills are still pretty weak so not sure where to go from here.
~‘In-Cluster’ Ceph - (Storage | Ubuntu) is not my favorite option since it requires a large number of hosts, but I gave it a shot. Got stuck because I didn’t understand the storage constraint requirements and the OSD nodes never deploy - resolved to “storage=root:0,4:64,5:8 zone=default”. I have VMs ready with (1)128GB and (1)16GB drives each, but evidently I’m still missing something.
~NFS-remote - (remote nfs | Juju) also seems like a good option for testing, but I’m not sure if this charm is even meant for dynamic provisioning at all.
~‘In-Cluster’ NFS - (Deploy Nfs using Charmhub - The Open Operator Collection) - Storage class and provisioner are created, but still doesn’t create PV’s from PVC’s. I’ve found some references this could be related to a Juju\LXD\Apparmor configuration (Bug #1251619 “nfs charm fails in install hook” : Bugs : nfs package : Juju Charms Collection) but I’ve tried the suggested fix with no success.
~StorageOS - (https://juju.is/tutorials/deploying-storageos-on-kubernetes#1-overview) - Doesn’t fully deploy daemonsets: Failed to pull image “gcr.io/google_containers/hyperkube:v1.20.2”
VM\KVM PROVISIONING
Right now the process of creating a pool of PVE VM’s for MaaS to manage is semi-manual:
-create VM in PVE, no boot media.
-spin up and wait for listing in MaaS.
-set Proxmox power driver settings in MaaS.
-commission machine in MaaS. (Curtin installs qemu-guest-agent)
I can create new PVE VMs in Ansible, and I can probably figure out how to configure the Proxmox powerdrivers with MaaS CLI, but at the moment manually spinning up new VMs is not an issue so I’ll worry about automating this part later.
I have been wondering if there would be any advantage to using KVM ‘pods’. Kind of seems like an extra layer the way I have things set up, but if it somehow facilitates better integration with Proxmox I’m open to it.
So that’s where I’m at. Planning to take another stab at ceph and nfs to see if I learn something new.
Found a little more info on the ceph-proxy issue I’m seeing.
I think it’s a known issue but doesn’t look like there’s anyone working on it.
Not sure if there’s a way to set that required ceph-storage.available flag through an overlay or something.
