Proper shutdown and restart instructions for Charmed Kubeflow?

#1

I installed and successfully run Kubeflow on microk8s using juju. However, I forgot to shut it down (and how anyway?) and when my machine came out of hibernate I wasn’t sure what to do. The only info about restarting juju I managed to find online dated back 2013? Could you please point me to the updated doc on this topic? Kubeflow components are coming up as microk8s resumes but seem struggling. Juju command wasn’t working.

It would be great if someone can point me to the proper shutdown and restart/resume procedures. Not sure if that matters but I am running microk8s uin Windows 11 WSL2 with Ubuntu.

#2

Hey Tim. The unresponsiveness of juju for me happened when I was on a machine without enough memory.

You’re running the entire Kubeflow on microk8s inside of WSL? How much memory do you have?

#3

32GB, and I used Kind before for kubeflow (much more overhead I believe) and no problem.

I just stopped and started microk8s again and all pods seems running. But juju status is not returning. Just hung there.

Bashtop showing 3 GB out of 16 being used.

#4

Apologies. I mean how much disk space?

#5

About 800GB free space.

Btw, juju eventually works again on my second microk8s stop/start.

However, a new issue: All my kubeflow pods are in running states. But juju reported the following under Unit: katib-db-manager/0* error idle 10.1.2.250 hook failed: “update-status” and katib-db-manager is waiting (“Installing agent”)

And the browser says cannot reach http://10.64.140.43.nip.io/ . Though I don’t think this is related to the katib error.

I have rerun the the dex-auth and oidc-gatekeeper uri configs (both got warning of already set). Did not help.

#6

I killed the katib-db-manager-0 pod and it recovered. Everything in green state now. I am good.

The browser access issue was user error. Please ignore :stuck_out_tongue:

So to conclude, there is no special stop and restart of Charmed Kubeflow is needed? Just restart microk8s and if some pods persistently fail just knock them down and let k8s to figure it out?

#7

As I understand it, yeah that’s how you do it :slight_smile: