Charmed Kubeflow 1.7 beta is here!

ca-scribner · 15 March 2023 11:54

Not yet, but there will be one when we make the stable release

Moula · 16 March 2023 09:40

hi, I tested this new version on my platform. Everything works except like version 1.6, Istio problem is still there . Needs commands to fix it.

munteanuandreea · 16 March 2023 10:10

Thank you Moula! The team is working now on the upgrade path for Istio. How did you find the deployment experience of 1.7 Beta?

Moula · 17 March 2023 08:40

Hi Andreea, The deployment works well except istio which always looks for the loadbalancer ip but never finds it

ca-scribner · 17 March 2023 12:30

Hi @Moula,

That looks like whatever is providing your load balancers in k8s might be malfunctioning. I see you’re using microk8s, can you check kubectl get pods -A and look at the metallb pods? I have a feeling something is wrong there.

Did you deploy this in the past day or two? microk8s historically pulled metallb images from dockerhub, but metallb’s dockerhub has been shut down just this week (see this for some context if interested). If your metallb pods are having image pull problems and their images are pointing to dockerhub, I think this is your problem. To work around this, microk8s 1.24/edge points to the up to date metallb repo, and that’ll hit other microk8s risks soon. Or, if you want to recover your current deployment, you should also be able to edit the metallb deployments and update the images like here

Moula · 17 March 2023 13:52

Hi @ca-scribner I’m not the only one having this problem. See: https://github.com/canonical/bundle-kubeflow/issues/559. The problem comes from the image of Metallb as you say.

I did the same deployment of my microk8s-ha cluster as with kubeflow 1.6 which worked after a few modifications. The deployment I redid it twice today and there is always the same problem.

ca-scribner · 17 March 2023 14:04

Hey @Moula,

Yeah I too was frustrated with this yesterday I feel your pain.

microk8s 1.24/edge works for me. Does that help? Manually changing those metallb images should also get it working, I just haven’t done that yet myself to solve things

Moula · 17 March 2023 14:22

@ca-scribner Thank you very much. I will try with cluster microk8s 1.24/edge this evening .

ca-scribner · 17 March 2023 14:29

welcome! Sorry for the frustration. Let us know how it goes

By the time you try it, things might have moved through the release ci too. atm I see

snap info microk8s
(truncated)
  1.24/stable:           v1.24.10        2023-02-02 (4561) 224MB classic
  1.24/candidate:        v1.24.10        2023-01-27 (4561) 224MB classic
  1.24/beta:             v1.24.10        2023-01-27 (4561) 224MB classic
  1.24/edge:             v1.24.11        2023-03-16 (4891) 225MB classic

where that v1.24.11 (rev 4891) works for me. Check that out before you install in case it has promoted through to the others already

Moula · 21 March 2023 10:30

@ca-scribner There is a new problem with knative-eventing

ca-scribner · 21 March 2023 13:25

Hi @Moula,

From the messages in the status of the units, I think you need to juju trust these charms - that should get things going. Did you deploy this from one of the premade bundles, or did you add these in separately? If a premade bundle, please let me know which one as we might have missed adding trust somewhere in our bundles.

Moula · 21 March 2023 18:18

@ca-scribner I just redid the deployment by bundle. The problem is still there. We must add trust to the charm :

ca-scribner · 21 March 2023 18:30

Thanks @moula, just so I don’t get it wrong can you tell me which bundle you’re using? Is the command juju deploy kubeflow --channel 1.7/beta?

Moula · 21 March 2023 18:31

@ca-scribner juju deploy kubeflow --channel 1.7/beta --trust

dnplas · 22 March 2023 11:10

Hi @Moula , thanks for reporting this. You can resolve this issue by running juju trust knative-eventing --scope=cluster, and the same for knative-serving. We have merged the fix for that already, you should not run into this issue in latest kubeflow (1.7/beta revision 333). Please let us know if you find any other issue.

ichvets · 22 March 2023 13:48

Just confirmed that latest (revision 333) deploys all components: juju deploy kubeflow --channel 1.7/beta --trust

Moula · 22 March 2023 15:22

Hi @dnplas just an another error message with : tensorboard-controller Thank’s.

dnplas · 22 March 2023 15:34

@Moula that is a known issue, the workaround for it is listed here.

dduala · 24 March 2023 04:55

I have successfully installed charmed-kubeflow on GCP but I don’t have access to It, how do I have access to it? (newby) I did run the following code already but no success:

juju config dex-auth public-url=http://<IP address>
juju config oidc-gatekeeper public-url=http://<IP address>

with my istio-ingressgateway ClusterIP

ca-scribner · 24 March 2023 19:43

Hey!

I’ve never tried myself on GCP, so I’m guessing a bit…

I’m guessing you’ve done the Configure the components and login to charmed kubeflow sections of the tutorial, but no luck? Theres an “expand to troubleshoot” under that section too that might help.

Are you trying to access the dashboard from your local machine but it is hosted on the GCP machine? You might need something like a SOCKS proxy to help out. This section talks about that.

Hope these help. Please let me know!