I have a 6 node Microk8s cluster with charmed kubeflow deployed.
I performed some network maintenance which meant the VMs with Microk8s were briefly disconnected for around 5 minutes from the network. After this they had some connectivity issues so I restarted the nodes. After restarting all 6 nodes Microk8s comes up succesfully and forms a quorum yet everything deployed in the Kubeflow namespace has an Unknown state. I’ve seen some old articles from 2019 where users where having a similar issue but it was patched so keen to see if anyone has any suggestions.
Please find my inspection report here: https://plik.root.gg/file/ldNHKrHbG2nTXnEV/q3CxtQdiM8iv0QrV/inspection-report-20220923_092917.tar.gz