Using GPUs with Charmed Kubeflow + Microk8s is not as easy as it is supposed to be, as Charmed Kubeflow only supports microk8s 1.21 or older at the moment.
My workaround was to change the container runtime, so as the microk8s installation would use the newest libnvidia-container version with the relevant bug fixes.
It is not really fault of the Charmed Kubeflow project, as it is due to the issues in the Kubeflow project and libnvidia-container, but I think this must be documented somewhere.
Thanks for your message, and sorry you’re having challenges with that. I agree, what you’re aiming for is not as straight forward as it should be.
One small comfort I have for you is that we and some others in the Kubeflow community have been advocating for Kubeflow to add support for more recent Kubernetes versions (there were some key APIs removed in 1.22 that have been in the way). It is highly likely that the next Kubeflow release (1.6, ~July) supports Kubernetes>1.21, and probable that we can even do a patch sooner (the only component that is still a problem is Notebooks, and they are targeting an out-of-band release sometime before Kubeflow 1.6.
All that said, it doesn’t help you right now. I’ll ask around to see if anyone internally has done this recently and has we have something we could turn into a blog post. If you have any more detail you’d like to share, that would also be helpful!
