kubeflow - Error from server (NotFound): services "istio-ingressgateway" not found

I follow Quick start guide to Kubeflow | Documentation | Charmed Kubeflow to juju deploy kubeflow --trust. However, istio-ingressgateway service is not found anywhere in the kubeflow namespace!

$ k get all -n kubeflow |grep istio-ingressgateway
pod/istio-ingressgateway-operator-0           1/1     Running           0          10m
service/istio-ingressgateway-operator        ClusterIP   10.152.183.126   <none>        30666/TCP                                10m
statefulset.apps/istio-ingressgateway-operator        1/1     10m
$ k -n kubeflow get svc istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
Error from server (NotFound): services "istio-ingressgateway" not found
$ juju status
Model     Controller          Cloud/Region        Version  SLA          Timestamp
kubeflow  microk8s-localhost  microk8s/localhost  2.9.32   unsupported  16:02:54+08:00

App                        Version                    Status   Scale  Charm                    Channel      Rev  Address         Exposed  Message
admission-webhook          res:oci-image@fc124ea      waiting      1  admission-webhook        stable        12                  no       waiting for container
argo-controller            res:oci-image@0eec3c1      waiting      1  argo-controller          stable        55                  no       
argo-server                res:oci-image@2eb4a20      active       1  argo-server              stable         5                  no       
dex-auth                   res:oci-image@a74f783      waiting      1  dex-auth                 2.28/stable   78                  no       waiting for container
envoy                      res:oci-image@b4adee5      waiting      1  envoy                    stable         6  10.152.183.25   no       
istio-ingressgateway                                  waiting      1  istio-gateway            1.5/stable    40                  no       Waiting for Istio Pilot information
istio-pilot                res:oci-image@87fc646      waiting      1  istio-pilot              1.5/stable    61  10.152.183.232  no       
jupyter-controller         res:oci-image@62a1ccf      waiting      1  jupyter-controller       stable        61                  no       waiting for container
jupyter-ui                 res:oci-image@5536a2d      active       1  jupyter-ui               stable        21  10.152.183.206  no       
katib-controller           res:oci-image@d8f2376      active       1  katib-controller         stable        40  10.152.183.85   no       
katib-db                   mariadb/server:10.3        active       1  charmed-osm-mariadb-k8s  stable        35  10.152.183.81   no       ready
katib-db-manager           res:oci-image@bee4154      waiting      1  katib-db-manager         stable        14  10.152.183.198  no       
katib-ui                   res:oci-image@2183b12      waiting      1  katib-ui                 stable        38  10.152.183.226  no       waiting for container
kfp-api                    res:oci-image@81e784a      waiting      1  kfp-api                  stable        33  10.152.183.142  no       
kfp-db                     mariadb/server:10.3        active       1  charmed-osm-mariadb-k8s  stable        35  10.152.183.107  no       ready
kfp-persistence            res:oci-image@1012943      waiting      1  kfp-persistence          stable        29                  no       
kfp-profile-controller     res:oci-image@14ec522      waiting      1  kfp-profile-controller   stable        16  10.152.183.252  no       
kfp-schedwf                res:oci-image@34e7e9e      active       1  kfp-schedwf              stable        32                  no       
kfp-ui                     res:oci-image@b67a29c      waiting      1  kfp-ui                   stable        32  10.152.183.110  no       waiting for container
kfp-viewer                 res:oci-image@c208ebd      waiting      1  kfp-viewer               stable        31                  no       
kfp-viz                    res:oci-image@13c46cf      waiting      1  kfp-viz                  stable        28  10.152.183.131  no       waiting for container
kubeflow-dashboard         res:oci-image@858a90f      error        1  kubeflow-dashboard       stable        64                  no       creating or updating custom resources: getting custom resources: attempt count exceeded: getting custom resource definition "profiles.kubeflow.org": custom resource definition "profiles.kubeflow.org" not found
kubeflow-profiles          res:profile-image@f4450cf  waiting      1  kubeflow-profiles        stable        57                  no       waiting for container
kubeflow-roles                                        active       1  kubeflow-roles           stable         1  10.152.183.176  no       
kubeflow-volumes           res:oci-image@fedee0e      waiting      1  kubeflow-volumes         stable        11  10.152.183.61   no       
metacontroller-operator                               active       1  metacontroller-operator  stable         2  10.152.183.95   no       
minio                      res:oci-image@1755999      waiting      1  minio                    stable        57  10.152.183.251  no       
mlmd                       res:oci-image@e2cb9ce      waiting      1  mlmd                     stable        10  10.152.183.116  no       
oidc-gatekeeper            res:oci-image@4e7f8dd      waiting      1  oidc-gatekeeper          stable        57  10.152.183.68   no       
seldon-controller-manager  res:oci-image@047f2fc      waiting      1  seldon-core              stable        52  10.152.183.136  no       waiting for container
tensorboard-controller     res:oci-image@0f8c7de      waiting      1  tensorboard-controller   stable         3  10.152.183.40   no       
tensorboards-web-app       res:oci-image@0c2babc      waiting      1  tensorboards-web-app     stable         3  10.152.183.203  no       
training-operator                                     active       1  training-operator        stable         6  10.152.183.121  no       

Unit                          Workload  Agent  Address       Ports                                   Message
admission-webhook/0*          waiting   idle                                                         waiting for container
argo-controller/0*            waiting   idle   10.1.208.11                                           waiting for container
argo-server/0*                active    idle   10.1.207.244  2746/TCP                                
dex-auth/0*                   waiting   idle                                                         waiting for container
envoy/0*                      waiting   idle   10.1.208.12   9901/TCP,9090/TCP                       waiting for container
istio-ingressgateway/0*       waiting   idle                                                         Waiting for Istio Pilot information
istio-pilot/0*                waiting   idle   10.1.207.211  8080/TCP,15010/TCP,15012/TCP,15017/TCP  waiting for container
jupyter-controller/0*         waiting   idle                                                         waiting for container
jupyter-ui/0*                 active    idle   10.1.207.218  5000/TCP                                
katib-controller/0*           active    idle   10.1.207.241  443/TCP,8080/TCP                        
katib-db-manager/0*           waiting   idle   10.1.208.5    6789/TCP                                waiting for container
katib-db/0*                   active    idle   10.1.207.196  3306/TCP                                ready
katib-ui/0*                   waiting   idle   10.1.208.1    8080/TCP                                waiting for container
kfp-api/0*                    waiting   idle   10.1.208.14   8888/TCP,8887/TCP                       waiting for container
kfp-db/0*                     active    idle   10.1.207.234  3306/TCP                                ready
kfp-persistence/0*            waiting   idle   10.1.208.9                                            waiting for container
kfp-profile-controller/0*     waiting   idle   10.1.208.10   80/TCP                                  waiting for container
kfp-schedwf/0*                active    idle   10.1.207.198                                          
kfp-ui/0*                     waiting   idle   10.1.208.13   3000/TCP                                waiting for container
kfp-viewer/0*                 waiting   idle   10.1.207.201                                          waiting for container
kfp-viz/0*                    waiting   idle   10.1.208.0    8888/TCP                                waiting for container
kubeflow-dashboard/0*         waiting   idle                                                         waiting for container
kubeflow-profiles/0*          waiting   idle                                                         waiting for container
kubeflow-roles/0*             active    idle   10.1.207.231                                          
kubeflow-volumes/0*           waiting   idle   10.1.208.2    5000/TCP                                waiting for container
metacontroller-operator/0*    active    idle   10.1.207.217                                          
minio/0*                      waiting   idle   10.1.208.7    9000/TCP                                waiting for container
mlmd/0*                       waiting   idle   10.1.208.15   8080/TCP                                waiting for container
oidc-gatekeeper/0*            waiting   idle   10.1.208.22   8080/TCP                                waiting for container
seldon-controller-manager/0*  waiting   idle                                                         waiting for container
tensorboard-controller/0*     waiting   idle   10.1.208.19   9443/TCP                                waiting for container
tensorboards-web-app/0*       waiting   idle   10.1.208.21   5000/TCP                                waiting for container
$ k version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.0-2+59bbb3530b6769", GitCommit:"59bbb3530b6769e4935a05ac0e13c9910c79253e", GitTreeState:"clean", BuildDate:"2022-05-13T06:43:45Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.0-2+59bbb3530b6769", GitCommit:"59bbb3530b6769e4935a05ac0e13c9910c79253e", GitTreeState:"clean", BuildDate:"2022-05-13T06:41:13Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

hey @kheteh, this seems like a deployment that hasn’t fully completed yet. Is this still in the same state after you waited some more time? this might be also an issue of resources, what is the size of the machine you are running this on?

Yes, still in this bad state. What do you mean “size of the machine”?

Hello @kheteh, it looks like you might be running Kubernetes 1.24. Unfortunately we only support Kubernetes 1.21 at the moment, although we are working hard to release support for a newer version of Kubernetes soon.

1 Like