Certificate Error When Adding Kubeflow via juju

I’m blindly following the installation documentation to get Microk8s installed and configured with kubeflow

But I’m hitting an error like this below:

joesan@joesan-InfinityBook-S-14-v5:~$ juju add-model kubeflow
ERROR opening API connection: starting proxy for api connection: connecting k8s proxy: Get "https://192.168.0.35:16443/api/v1/namespaces/controller-microk8s-localhost/services/controller-service": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "10.152.183.1")

Looks like some certificate does not add up, but how do I get past this? I’m sure this is something very simple that I’m missing, but the documentation does not say anything about this. Any ideas how to fix this?

joesan@joesan-InfinityBook-S-14-v5:~$ juju controllers
Use --refresh option with this command to see the latest information.

Controller           Model     User   Access     Cloud/Region        Models  Nodes  HA  Version
microk8s-localhost*  kubeflow  admin  superuser  microk8s/localhost       2      1   -  2.9.28
1 Like

If you run something like microk8s.kubectl get ns does it return anything?

joesan@joesan-InfinityBook-S-14-v5:~$ microk8s.kubectl get ns
NAME              STATUS   AGE
kube-system       Active   10h
kube-public       Active   10h
kube-node-lease   Active   10h
default           Active   10h
ingress           Active   10h
metallb-system    Active   10h
joesan@joesan-InfinityBook-S-14-v5:~$

joesan@joesan-InfinityBook-S-14-v5:~$ juju clouds --client
Only clouds with registered credentials are shown.
There are more clouds, use --all to see them.
You can bootstrap a new controller using one of these clouds...

Clouds available on the client:
Cloud      Regions  Default    Type  Credentials  Source    Description
localhost  1        localhost  lxd   0            built-in  LXD Container Hypervisor
microk8s   1        localhost  k8s   1            built-in  A Kubernetes Cluster

joesan@joesan-InfinityBook-S-14-v5:~$

Removing

~$ .local/share/juju

did the trick for me. There were some stale credentials lying around from a previous installation I guess.

1 Like

Thanks for the follow-up, @joesan. If you want to avoid removing the entire dir in the future, you should be able to clear just the offending credentials by editing ~/.local/share/juju/credentials.yaml and ~/.local/share/juju/clouds.yaml.

@jameinel it’s fairly common to run into stale credentials when working against dev controllers, because they get stood up and torn down so often. Should we open a ticket w/ the Juju client, requesting that we better detect and handle the situation?

For me, it breaks 100% reproducibly when I try to create an offer against a microk8s controller.

@michele-mancioppi do you mean that you are blocked from creating cross model relations for microk8s? Or do you mean that you need to edit the credentials whenever creating one?

I mean that very often, whenever I execute a juju offer …, something happens to my credentials and it borks in a way that looks the same as reported by joesan. It’s really weird.

So we already have juju unregister for the controller credentials. Is the issue that they replaced microk8s itself underneath us and then we complain about the cloud credentials themselves?