Error Getting Started with Juju & MicroK8s

juju
1

I’ve been trying to get started by following the "Using Juju with MicroK8s instructions, but get stuck bootstrapping a juju controller:

https://juju.is/docs/microk8s-cloud

I follow all the instructions, but the juju bootstrap microk8s micro command inevitably returns this:

$ juju bootstrap microk8s micro
Creating Juju controller "micro" on microk8s/localhost
Fetching Juju Dashboard 0.3.0
Creating k8s resources for controller "controller-micro"
Starting controller pod
Bootstrap agent now started
Contacting Juju controller at 10.152.183.233 to verify accessibility...
ERROR unable to contact api server after 1 attempts: unable to connect to API: dial tcp 10.152.183.233:17070: connect: connection refused

Have completely wiped the install and tried again multiple times with the same results.

Looking at the mongodb log in /var/log/containers, it has these errors:

2020-11-17T13:59:04.624783383-08:00 stdout F 2020-11-17T21:59:04.624+0000 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2020-11-17T13:59:04.625982877-08:00 stdout F 2020-11-17T21:59:04.625+0000 E NETWORK  [main] cannot read certificate file: /var/lib/juju/server.pem error:02001002:system library:fopen:No such file or directory
2020-11-17T13:59:04.626097973-08:00 stdout F 2020-11-17T21:59:04.626+0000 F CONTROL  [main] Failed global initialization: InvalidSSLConfiguration: Can not set up PEM key file.

Not sure if it’s related or not?

Any ideas? This is an Ubuntu 20.10 desktop system.

2

I just tested this by starting a totally fresh 20.10 instance.

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.10
DISTRIB_CODENAME=groovy
DISTRIB_DESCRIPTION="Ubuntu 20.10"
$ sudo snap install microk8s --classic
$ sudo usermod -a -G microk8s $USER
logout and login
$ sudo snap install juju --classic
$ microk8s.enable dns storage
$ juju bootstrap microk8s test

The TLS 1.0 message is just a warning from mongodb that it is not using the insecure TLS 1.0 protocol.

Can you try bootstrapping with --debug… maybe there will be something helpful to indicate what’s wrong.

3

I’m experiencing the same problem on Ubuntu 20.04, following the microk8s install instructions. Controller creation appears to go well until the “attempt to verify accessibility”:

...
14:26:04 DEBUG juju.kubernetes.provider bootstrap.go:884 Successfully assigned controller-mk8s-ctlr/controller-0 to twelf
14:26:04 DEBUG juju.kubernetes.provider bootstrap.go:884 Pulled images
14:26:04 DEBUG juju.kubernetes.provider events.go:52 getting the latest event for "involvedObject.name=controller-0,involvedObject.kind=Pod"
14:26:04 DEBUG juju.kubernetes.provider.watcher k8swatcher.go:115 fire notify watcher for controller-0
14:26:05 DEBUG juju.kubernetes.provider events.go:52 getting the latest event for "involvedObject.name=controller-0,involvedObject.kind=Pod"
14:26:05 DEBUG juju.kubernetes.provider.watcher k8swatcher.go:115 fire notify watcher for controller-0
14:26:05 DEBUG juju.kubernetes.provider bootstrap.go:884 Created container mongodb
14:26:05 DEBUG juju.kubernetes.provider bootstrap.go:884 Started mongodb container
14:26:07 DEBUG juju.kubernetes.provider.watcher k8swatcher.go:115 fire notify watcher for controller-0
14:26:07 DEBUG juju.kubernetes.provider events.go:52 getting the latest event for "involvedObject.name=controller-0,involvedObject.kind=Pod"
14:26:07 DEBUG juju.kubernetes.provider bootstrap.go:884 Created container api-server
14:26:07 DEBUG juju.kubernetes.provider bootstrap.go:884 Started controller container
14:26:07 DEBUG juju.kubernetes.provider.watcher k8swatcher.go:115 fire notify watcher for controller
14:26:07 DEBUG juju.kubernetes.provider events.go:52 getting the latest event for "involvedObject.name=controller-0,involvedObject.kind=Pod"
14:26:07 INFO  cmd bootstrap.go:972 Starting controller pod
14:26:07 INFO  cmd bootstrap.go:669 Bootstrap agent now started
14:26:07 DEBUG juju.kubernetes.provider events.go:52 getting the latest event for "involvedObject.name=controller,involvedObject.kind=StatefulSet"
14:26:07 INFO  juju.juju api.go:330 API endpoints changed from [] to [10.152.183.215:17070]
14:26:07 INFO  cmd controller.go:93 Contacting Juju controller at 10.152.183.215 to verify accessibility...
14:26:07 INFO  juju.juju api.go:78 connecting to API addresses: [10.152.183.215:17070]
14:27:07 INFO  cmd controller.go:135 Still waiting for API to become available: starting proxy for api connection: waiting for pod controller-0 to become ready for tunnel: context deadline exceeded
14:27:07 INFO  juju.juju api.go:78 connecting to API addresses: [10.152.183.215:17070]
14:28:07 INFO  cmd controller.go:135 Still waiting for API to become available: starting proxy for api connection: waiting for pod controller-0 to become ready for tunnel: context deadline exceeded
14:28:07 INFO  juju.juju api.go:78 connecting to API addresses: [10.152.183.215:17070]
14:29:07 INFO  cmd controller.go:135 Still waiting for API to become available: starting proxy for api connection: waiting for pod controller-0 to become ready for tunnel: context deadline exceeded
14:29:07 INFO  juju.juju api.go:78 connecting to API addresses: [10.152.183.215:17070]
14:30:07 INFO  cmd controller.go:135 Still waiting for API to become available: starting proxy for api connection: waiting for pod controller-0 to become ready for tunnel: context deadline exceeded
14:30:07 INFO  juju.juju api.go:78 connecting to API addresses: [10.152.183.215:17070]
14:31:07 INFO  cmd controller.go:135 Still waiting for API to become available: starting proxy for api connection: waiting for pod controller-0 to become ready for tunnel: context deadline exceeded
14:31:07 INFO  juju.juju api.go:78 connecting to API addresses: [10.152.183.215:17070]
14:32:07 INFO  cmd controller.go:135 Still waiting for API to become available: starting proxy for api connection: waiting for pod controller-0 to become ready for tunnel: context deadline exceeded
14:32:07 INFO  juju.juju api.go:78 connecting to API addresses: [10.152.183.215:17070]
<hit ^C at this point to cancel and cleanup>
4

Additional information: the log file from the failing pod0:

[twelf:juju () 263] => kc logs -n controller-mk8s-ctlr pod/controller-0 mongodb
2021-06-24T22:26:06.591+0000 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2021-06-24T22:26:06.593+0000 E NETWORK  [main] cannot read certificate file: /var/lib/juju/server.pem error:02001002:system library:fopen:No such file or directory
2021-06-24T22:26:06.593+0000 F CONTROL  [main] Failed global initialization: InvalidSSLConfiguration: Can not set up PEM key file.

Indeed, there is no such PEM file in the location being searched. Is this the right path? Should there me a PEM file there? If so, how is it created?

5

Is there any solution as I am also facing the same issue.
18:08:47 INFO cmd controller.go:135 Still waiting for API to become available: starting proxy for api connection: waiting for pod controller-0 to become ready for tunnel: context deadline exceeded
ERROR contacting controller (cancelled): starting proxy for api connection: waiting for pod controller-0 to become ready for tunnel: context deadline exceeded

6

So it looks like we only wait for 1 minute for the pod to be ready. I think this is a bit short, I’ll make a PR to improve the time out for this. This will hopefully land in next week’s release.

See: Bug #1934494 “microk8s tunnel: Waiting for pod to become ready t...” : Bugs : juju

1 Like
7

Is there a chance that you are running on btrfs? It may be worth enabling --feature-gates="LocalStorageCapacityIsolation=false" in the kubelet config (check the microk8s logs to see whether kubelet cannot establish the storage path)