Certificate Error When Adding Kubeflow via juju

getting started
#1

I’m blindly following the installation documentation to get Microk8s installed and configured with kubeflow

But I’m hitting an error like this below:

joesan@joesan-InfinityBook-S-14-v5:~$ juju add-model kubeflow
ERROR opening API connection: starting proxy for api connection: connecting k8s proxy: Get "https://192.168.0.35:16443/api/v1/namespaces/controller-microk8s-localhost/services/controller-service": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "10.152.183.1")

Looks like some certificate does not add up, but how do I get past this? I’m sure this is something very simple that I’m missing, but the documentation does not say anything about this. Any ideas how to fix this?

joesan@joesan-InfinityBook-S-14-v5:~$ juju controllers
Use --refresh option with this command to see the latest information.

Controller           Model     User   Access     Cloud/Region        Models  Nodes  HA  Version
microk8s-localhost*  kubeflow  admin  superuser  microk8s/localhost       2      1   -  2.9.28
#2

If you run something like microk8s.kubectl get ns does it return anything?

#3 
joesan@joesan-InfinityBook-S-14-v5:~$ microk8s.kubectl get ns
NAME              STATUS   AGE
kube-system       Active   10h
kube-public       Active   10h
kube-node-lease   Active   10h
default           Active   10h
ingress           Active   10h
metallb-system    Active   10h
joesan@joesan-InfinityBook-S-14-v5:~$

joesan@joesan-InfinityBook-S-14-v5:~$ juju clouds --client
Only clouds with registered credentials are shown.
There are more clouds, use --all to see them.
You can bootstrap a new controller using one of these clouds...

Clouds available on the client:
Cloud      Regions  Default    Type  Credentials  Source    Description
localhost  1        localhost  lxd   0            built-in  LXD Container Hypervisor
microk8s   1        localhost  k8s   1            built-in  A Kubernetes Cluster

joesan@joesan-InfinityBook-S-14-v5:~$
#4

Removing

~$ .local/share/juju

did the trick for me. There were some stale credentials lying around from a previous installation I guess.

1 Like
#5

Thanks for the follow-up, @joesan. If you want to avoid removing the entire dir in the future, you should be able to clear just the offending credentials by editing ~/.local/share/juju/credentials.yaml and ~/.local/share/juju/clouds.yaml.

@jameinel it’s fairly common to run into stale credentials when working against dev controllers, because they get stood up and torn down so often. Should we open a ticket w/ the Juju client, requesting that we better detect and handle the situation?

#6

For me, it breaks 100% reproducibly when I try to create an offer against a microk8s controller.

#7

@michele-mancioppi do you mean that you are blocked from creating cross model relations for microk8s? Or do you mean that you need to edit the credentials whenever creating one?

#8

I mean that very often, whenever I execute a juju offer …, something happens to my credentials and it borks in a way that looks the same as reported by joesan. It’s really weird.

#9

So we already have juju unregister for the controller credentials. Is the issue that they replaced microk8s itself underneath us and then we complain about the cloud credentials themselves?