ERROR resolving microk8s credentials: ensuring cluster role "juju-credential-microk8s" in namespace "kube-system"

juju
, ,
1

I try to reproduce aquickstart guide for running Kubeflow on my Laptop. All steps was done correctly until the following one:

aimaster@aimaster001:~$ juju bootstrap microk8s

ERROR resolving microk8s credentials: ensuring cluster role “juju-credential-microk8s” in namespace “kube-system”: Get https://172.16.32.232:16443/apis/rbac.authorization.k8s.io/v1/clusterroles/juju-credential-microk8s": Forbidden

Any idea what am I doing wrong? Here is my setup: I am running following setup: Asus Zenbook Pro UX582LR, X86_64 Architecture, Ubuntu 22.04 LTS

Info:

  • juju 2.9.35
  • juju-bundle 0.4.0
  • juju-wait 2.8.4~2.8.4
  • kubectl 1.25.3
  • microk8s v1.24.6

aimaster@aimaster001:~$ microk8s inspect
Inspecting system
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-kubelite is running
Service snap.microk8s.daemon-k8s-dqlite is running
Service snap.microk8s.daemon-apiserver-kicker is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting dqlite
Inspect dqlite

Building the report tarball
Report tarball is at /var/snap/microk8s/4023/inspection-report-
20221102_183658.tar.gz
aimaster@aimaster001:~$

aimaster@aimaster001:~$ microk8s status
microk8s is running
high-availability: no
datastore master nodes: 127.0.0.1:19001
datastore standby nodes: none
addons:
enabled: dashboard # (core) The Kubernetes dashboard
dns # (core) CoreDNS
ha-cluster # (core) Configure high availability on the current node
helm3 # (core) Helm 3 - Kubernetes package manager
hostpath-storage # (core) Storage class; allocates storage from host directory
ingress # (core) Ingress controller for external access
metallb # (core) Loadbalancer for your Kubernetes cluster
metrics-server # (core) K8s Metrics Server for API access to service metrics
storage # (core) Alias to hostpath-storage add-on, deprecated
disabled:
community # (core) The community addons repository
gpu # (core) Automatic enablement of Nvidia CUDA
helm # (core) Helm 2 - the package manager for Kubernetes
host-access # (core) Allow Pods connecting to Host services smoothly
mayastor # (core) OpenEBS MayaStor
prometheus # (core) Prometheus operator for monitoring and logging
rbac # (core) Role-Based Access Control for authorisation
registry # (core) Private image registry exposed on localhost:32000

2

I just tested myself with juju 2.9.35 snap and microk8s from the 1.24 track. There was no issue for me. I assume you can kubectl create k8s resources outside of Juju to confirm the micrk8s install is working as expected?

@kelvin.liu or @tlm have you seen this error before?

3

this is output with debug option

juju_issue
juju_issue1620×471 34.6 KB

4

Hey folks, I just ran into the same issue when bootstrapping an existing Microk8s deployment and found a solution. For completeness, here was my exact error:

$ juju bootstrap microk8s --debug
15:26:30 INFO  juju.cmd supercommand.go:56 running juju [2.9.45 afb8ee760af71d0bca8c3e4e0dc28af2dabc9b1d gc go1.20.8]
15:26:30 DEBUG juju.cmd supercommand.go:57   args: []string{"/snap/juju/24550/bin/juju", "bootstrap", "microk8s", "--debug"}
ERROR resolving microk8s credentials: ensuring cluster role "juju-credential-microk8s" in namespace "kube-system": Unauthorized
15:26:30 DEBUG cmd supercommand.go:537 error stack: 
Unauthorized
github.com/juju/juju/caas/kubernetes/clientconfig.getOrCreateClusterRole:167: 
github.com/juju/juju/caas/kubernetes/clientconfig.ensureJujuAdminServiceAccount:88: ensuring cluster role "juju-credential-microk8s" in namespace "kube-system"
github.com/juju/juju/caas/kubernetes/provider.attemptMicroK8sCredential:61: resolving microk8s credentials
github.com/juju/juju/caas/kubernetes/provider.environProviderCredentials.RegisterCredentials:78: 
github.com/juju/juju/cmd/juju/commands.(*bootstrapCommand).cloud:1146: 
github.com/juju/juju/cmd/juju/commands.(*bootstrapCommand).Run:624:

Tracking down the error in the code led me here, which mentions a group permissions issue. On my system, the snap_microk8s didn’t exist. So I created it and added my user:

sudo groupadd snap_microk8s
sudo usermod -aG snap_microk8s $USER

After logging out and in again to refresh group membership, I was able to successfully bootstrap.