Error while installing Kubeflow via the new tutorial

charm
1

Hi, I am trying to install Kubeflow locally using the new tutorial, but I am stuck at this step:

controller  microk8s-localhost  microk8s/localhost  2.9.22   unsupported  14:51:57Z

App                        Version                Status   Scale  Charm                    Store     Channel      Rev  OS          Address         Message
admission-webhook          res:oci-image@fc124ea  active       1  admission-webhook        charmhub  stable        12  kubernetes  10.152.183.44   
argo-controller                                   blocked      1  argo-controller          charmhub  stable        56  kubernetes                  Waiting for object-storage relation data
dex-auth                                          waiting    0/1  dex-auth                 charmhub  2.28/stable   78  kubernetes                  installing agent
envoy                                             waiting    0/1  envoy                    charmhub  stable         6  kubernetes                  installing agent
istio-ingressgateway                              waiting    0/1  istio-gateway            charmhub  1.5/stable    40  kubernetes                  installing agent
istio-pilot                                       waiting    0/1  istio-pilot              charmhub  1.5/stable    61  kubernetes                  installing agent
jupyter-controller                                waiting    0/1  jupyter-controller       charmhub  stable        61  kubernetes                  installing agent
jupyter-ui                                        waiting    0/1  jupyter-ui               charmhub  stable        21  kubernetes                  installing agent
kfp-api                                           waiting    0/1  kfp-api                  charmhub  stable        33  kubernetes                  installing agent
kfp-db                                            waiting    0/1  charmed-osm-mariadb-k8s  charmhub  stable        35  kubernetes                  installing agent
kfp-persistence                                   waiting    0/1  kfp-persistence          charmhub  stable        29  kubernetes                  installing agent
kfp-profile-controller                            waiting    0/1  kfp-profile-controller   charmhub  stable        16  kubernetes                  installing agent
kfp-schedwf                                       waiting    0/1  kfp-schedwf              charmhub  stable        32  kubernetes                  installing agent
kfp-ui                                            waiting    0/1  kfp-ui                   charmhub  stable        32  kubernetes                  installing agent
kfp-viewer                                        waiting    0/1  kfp-viewer               charmhub  stable        31  kubernetes                  installing agent
kfp-viz                                           waiting    0/1  kfp-viz                  charmhub  stable        28  kubernetes                  installing agent
kubeflow-dashboard                                waiting    0/1  kubeflow-dashboard       charmhub  stable        64  kubernetes                  installing agent
kubeflow-profiles                                 waiting    0/1  kubeflow-profiles        charmhub  stable        57  kubernetes                  installing agent
kubeflow-roles                                    waiting    0/1  kubeflow-roles           charmhub  stable         1  ubuntu      10.152.183.96   installing agent
kubeflow-volumes                                  waiting    0/1  kubeflow-volumes         charmhub  stable        11  kubernetes                  installing agent
metacontroller-operator                           waiting    0/1  metacontroller-operator  charmhub  stable         2  ubuntu      10.152.183.239  installing agent
minio                                             waiting    0/1  minio                    charmhub  stable        57  kubernetes                  installing agent
mlmd                                              waiting    0/1  mlmd                     charmhub  stable        10  kubernetes                  installing agent
oidc-gatekeeper                                   waiting    0/1  oidc-gatekeeper          charmhub  stable        57  kubernetes                  installing agent
seldon-controller-manager                         waiting    0/1  seldon-core              charmhub  stable        52  kubernetes                  installing agent
training-operator                                 waiting    0/1  training-operator        charmhub  stable         6  kubernetes  10.152.183.214  installing agent

Unit                         Workload  Agent       Address      Ports     Message
admission-webhook/0*         active    idle        10.1.188.12  4443/TCP  
argo-controller/0*           blocked   idle                               Waiting for object-storage relation data
dex-auth/0                   waiting   allocating                         installing agent
envoy/0                      waiting   allocating                         installing agent
istio-ingressgateway/0       waiting   allocating                         installing agent
istio-pilot/0                waiting   allocating                         installing agent
jupyter-controller/0         waiting   allocating                         installing agent
jupyter-ui/0                 waiting   allocating                         installing agent
kfp-api/0                    waiting   allocating                         installing agent
kfp-db/0                     waiting   allocating                         installing agent
kfp-persistence/0            waiting   allocating                         installing agent
kfp-profile-controller/0     waiting   allocating                         installing agent
kfp-schedwf/0                waiting   allocating                         installing agent
kfp-ui/0                     waiting   allocating                         installing agent
kfp-viewer/0                 waiting   allocating                         installing agent
kfp-viz/0                    waiting   allocating                         installing agent
kubeflow-dashboard/0         waiting   allocating                         installing agent
kubeflow-profiles/0          waiting   allocating                         installing agent
kubeflow-roles/0             waiting   allocating                         installing agent
kubeflow-volumes/0           waiting   allocating                         installing agent
metacontroller-operator/0    waiting   allocating                         installing agent
minio/0                      waiting   allocating                         installing agent
mlmd/0                       waiting   allocating                         installing agent
oidc-gatekeeper/0            waiting   allocating                         installing agent
seldon-controller-manager/0  waiting   allocating                         installing agent
training-operator/0          waiting   allocating                         installing agent
2

Hello! Can you check if you have sufficient capacity in your cluster to launch the various pods and PVCs?

What do these commands output?

kubectl get pvc -nkubeflow
kubectl get pods -nkubeflow

Please also use the kubectl describe -nkubeflow command to have a look at the status of the pods and PVCs in more detail, and see if they are blocked waiting for resources, or if there is some other problem (eg. firewall preventing the container images downloading or similar).

Thanks

3

Something might have gone wrong with the admission-webhook. If it isn’t working, it will block pod creation. You should be able to see that through kubectl get pods -n kubeflow and then kubectl describe pods SOME_PENDING_POD. You can do the same for the admission-webhook pod (and also kubectl logs on the admission-webhook) to see if there’s anything of note there.

Have you already deployed kubeflow once in this instance of microk8s? Sometimes things can be left over when removing - an option could be to sudo snap remove microk8s --purge and reinstall to see if that helps.

4

According to your juju status output, you are on the controller model. Docs mention that you need to create a model named “kubeflow”. Otherwise, the deployment will be stuck as you experienced.

Please create kubeflow model by using

juju add-model kubeflow

if it already exists switch to the related model by using the following command:

juju switch kubeflow

1 Like