Juju bootstrap fails with "forbidden"

1

I am trying to bootstrap a cluster to install k8s. I already added my MAAS cluster on juju and now I can’t bootstrap. Here is my output:

rabjen@controller:~$ juju bootstrap minilama --debug --config=config.yaml
09:42:16 INFO  juju.cmd supercommand.go:57 running juju [2.6.10 gc go1.10.4]
09:42:16 DEBUG juju.cmd supercommand.go:58   args: []string{"/snap/juju/9484/bin/juju", "bootstrap", "minilama", "--debug", "--config=config.yaml"}
09:42:16 DEBUG juju.cmd.juju.commands bootstrap.go:1038 authenticating with region "" and credential "rabjen" ()
09:42:16 DEBUG juju.cmd.juju.commands bootstrap.go:1166 provider attrs: map[]
09:42:18 INFO  cmd authkeys.go:114 Adding contents of "/home/rabjen/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
09:42:18 INFO  cmd authkeys.go:114 Adding contents of "/home/rabjen/.ssh/id_rsa.pub" to authorized-keys
09:42:18 DEBUG juju.cmd.juju.commands bootstrap.go:1225 preparing controller with config: map[type:maas max-action-results-size:5G transmit-vendor-metrics:true ftp-proxy: apt-no-proxy: agent-metadata-url: image-stream:released snap-store-proxy-url: agent-stream:released snap-store-proxy: container-image-stream:released test-mode:false no-proxy:127.0.0.1,localhost,::1 snap-http-proxy: juju-no-proxy:127.0.0.1,localhost,::1 uuid:4c1b9b28-f797-4a38-8ee2-969a6b117140 logforward-enabled:false juju-ftp-proxy: logging-config: net-bond-reconfigure-delay:17 development:false fan-config: resource-tags: apt-ftp-proxy: enable-os-upgrade:true http-proxy:http://192.168.42.250:3128 automatically-retry-hooks:true disable-network-management:false apt-mirror: snap-store-assertions: name:controller image-metadata-url: container-inherit-properties: ignore-machine-addresses:false https-proxy:http://192.168.42.250:3128 max-status-history-age:336h authorized-keys:ssh-rsa (redacted) juju-client-key
ssh-rsa (redacted) rabjen@controller
 proxy-ssh:false juju-https-proxy: juju-http-proxy: max-action-results-age:336h apt-https-proxy:http://192.168.42.250:3128 update-status-hook-interval:5m provisioner-harvest-mode:destroyed ssl-hostname-verification:true default-series:bionic cloudinit-userdata: enable-os-refresh-update:true container-networking-method: max-status-history-size:5G container-image-metadata-url: firewall-mode:instance backup-dir: egress-subnets: apt-http-proxy:http://192.168.42.250:3128 snap-https-proxy:]
09:42:18 DEBUG juju.provider.maas environprovider.go:62 opening model "controller".
09:42:18 INFO  cmd bootstrap.go:677 Creating Juju controller "minilama" on minilama
09:42:18 INFO  juju.cmd.juju.commands bootstrap.go:746 combined bootstrap constraints:
09:42:18 DEBUG juju.environs.bootstrap bootstrap.go:262 model "controller" supports application/machine networks: true
09:42:18 DEBUG juju.environs.bootstrap bootstrap.go:264 network management by juju enabled: true
09:42:18 INFO  cmd bootstrap.go:296 Loading image metadata
09:42:18 INFO  cmd bootstrap.go:373 Looking for packaged Juju agent version 2.6.10 for amd64
09:42:18 INFO  juju.environs.bootstrap tools.go:72 looking for bootstrap agent binaries: version=2.6.10
09:42:18 DEBUG juju.environs.tools tools.go:102 finding agent binaries in stream: "released"
09:42:18 DEBUG juju.environs.tools tools.go:104 reading agent binaries with major.minor version 2.6
09:42:18 DEBUG juju.environs.tools tools.go:112 filtering agent binaries by version: 2.6.10
09:42:18 DEBUG juju.environs.tools tools.go:118 filtering agent binaries by architecture: amd64
09:42:18 DEBUG juju.environs.tools urls.go:116 trying datasource "keystone catalog"
09:42:33 DEBUG juju.environs.simplestreams simplestreams.go:683 using default candidate for content id "com.ubuntu.juju:released:tools" are {20161007 mirrors:1.0 content-download streams/v1/cpc-mirrors.sjson []}
09:42:34 INFO  juju.environs.bootstrap tools.go:74 found 17 packaged agent binaries
09:42:34 INFO  cmd bootstrap.go:467 Starting new instance for initial controller
09:42:34 INFO  cmd bootstrap.go:161 Launching controller instance(s) on minilama...
09:42:34 DEBUG juju.provider.maas environ.go:960 attempting to acquire node in zone "default"
09:42:36 DEBUG juju.cloudconfig.instancecfg instancecfg.go:956 Setting numa ctl preference to false
09:42:36 DEBUG juju.service discovery.go:64 discovered init system "systemd" from series "bionic"
09:42:36 DEBUG juju.provider.maas environ.go:1051 maas user data; 3820 bytes
09:42:42 DEBUG juju.provider.maas environ.go:1097 started instance "4cxt8h"
09:42:42 INFO  cmd bootstrap.go:235  - 4cxt8h (arch=amd64 mem=16G cores=4)
09:42:42 INFO  juju.environs.bootstrap bootstrap.go:805 newest version: 2.6.10
09:42:42 INFO  juju.environs.bootstrap bootstrap.go:820 picked bootstrap agent binary version: 2.6.10
09:42:42 INFO  cmd bootstrap.go:509 Installing Juju agent on bootstrap instance
09:42:43 INFO  cmd bootstrap.go:626 Fetching Juju GUI 2.15.0
09:49:43 DEBUG juju.cloudconfig.instancecfg instancecfg.go:956 Setting numa ctl preference to false
Waiting for address
09:49:44 DEBUG juju.provider.maas maas2instance.go:87 "node1" has addresses ["local-cloud:192.168.42.101@undefined(id:-1)"]
Attempting to connect to 192.168.42.101:22
09:49:44 DEBUG juju.provider.common bootstrap.go:576 connection attempt for 192.168.42.101 failed: ssh: connect to host 192.168.42.101 port 22: Connection refused
09:49:49 DEBUG juju.provider.common bootstrap.go:576 connection attempt for 192.168.42.101 failed: ssh: connect to host 192.168.42.101 port 22: Connection refused
09:49:54 DEBUG juju.provider.common bootstrap.go:576 connection attempt for 192.168.42.101 failed: ssh: connect to host 192.168.42.101 port 22: Connection refused
09:49:54 DEBUG juju.provider.maas maas2instance.go:87 "node1" has addresses ["local-cloud:192.168.42.101@undefined(id:-1)"]
09:50:04 DEBUG juju.provider.common bootstrap.go:576 connection attempt for 192.168.42.101 failed: /var/lib/juju/nonce.txt does not exist
09:50:04 DEBUG juju.provider.maas maas2instance.go:87 "node1" has addresses ["local-cloud:192.168.42.101@undefined(id:-1)"]
09:50:10 DEBUG juju.provider.common bootstrap.go:576 connection attempt for 192.168.42.101 failed: /var/lib/juju/nonce.txt does not exist
09:50:14 DEBUG juju.provider.maas maas2instance.go:87 "node1" has addresses ["local-cloud:192.168.42.101@undefined(id:-1)"]
09:50:15 DEBUG juju.provider.common bootstrap.go:576 connection attempt for 192.168.42.101 failed: /var/lib/juju/nonce.txt does not exist
09:50:21 DEBUG juju.provider.common bootstrap.go:576 connection attempt for 192.168.42.101 failed: /var/lib/juju/nonce.txt does not exist
09:50:24 DEBUG juju.provider.maas maas2instance.go:87 "node1" has addresses ["local-cloud:192.168.42.101@undefined(id:-1)"]
09:50:27 DEBUG juju.provider.common bootstrap.go:576 connection attempt for 192.168.42.101 failed: /var/lib/juju/nonce.txt does not exist
09:50:32 DEBUG juju.provider.common bootstrap.go:576 connection attempt for 192.168.42.101 failed: /var/lib/juju/nonce.txt does not exist
09:50:34 DEBUG juju.provider.maas maas2instance.go:87 "node1" has addresses ["local-cloud:192.168.42.101@undefined(id:-1)"]
09:50:38 DEBUG juju.provider.common bootstrap.go:576 connection attempt for 192.168.42.101 failed: /var/lib/juju/nonce.txt does not exist
09:50:44 DEBUG juju.provider.common bootstrap.go:576 connection attempt for 192.168.42.101 failed: /var/lib/juju/nonce.txt does not exist
09:50:44 DEBUG juju.provider.maas maas2instance.go:87 "node1" has addresses ["local-cloud:192.168.42.101@undefined(id:-1)"]
09:50:49 DEBUG juju.provider.common bootstrap.go:576 connection attempt for 192.168.42.101 failed: /var/lib/juju/nonce.txt does not exist
09:50:54 DEBUG juju.provider.maas maas2instance.go:87 "node1" has addresses ["local-cloud:192.168.42.101@undefined(id:-1)"]
09:50:55 DEBUG juju.provider.common bootstrap.go:576 connection attempt for 192.168.42.101 failed: /var/lib/juju/nonce.txt does not exist
09:51:01 INFO  cmd bootstrap.go:345 Connected to 192.168.42.101
09:51:01 INFO  juju.cloudconfig userdatacfg_unix.go:537 Fetching agent: curl -sSfw 'agent binaries from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 --proxy http://192.168.42.250:3128 --noproxy 127.0.0.1,localhost,::1 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/agent/2.6.10/juju-2.6.10-ubuntu-amd64.tgz]>
09:51:01 INFO  cmd bootstrap.go:415 Running machine configuration script...
09:55:24 INFO  cmd bootstrap.go:564 Bootstrap agent now started
09:55:24 DEBUG juju.provider.maas maas2instance.go:87 "node1" has addresses ["local-cloud:192.168.42.101@undefined(id:-1)"]
09:55:24 INFO  juju.juju api.go:303 API endpoints changed from [] to [192.168.42.101:17070]
09:55:24 INFO  cmd controller.go:89 Contacting Juju controller at 192.168.42.101 to verify accessibility...
09:55:24 INFO  juju.juju api.go:67 connecting to API addresses: [192.168.42.101:17070]
10:05:22 ERROR juju.cmd.juju.commands bootstrap.go:697 unable to contact api server after 1 attempts: unable to connect to API: Forbidden
10:05:22 DEBUG juju.cmd.juju.commands bootstrap.go:698 (error details: [{/build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/common/controller.go:128: unable to contact api server after 1 attempts} {/build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/common/controller.go:44: } {/build/juju/parts/juju/go/src/github.com/juju/juju/juju/api.go:72: } {/build/juju/parts/juju/go/src/github.com/juju/juju/api/apiclient.go:207: } {/build/juju/parts/juju/go/src/github.com/juju/juju/api/apiclient.go:622: } {/build/juju/parts/juju/go/src/github.com/juju/juju/api/apiclient.go:967: } {/build/juju/parts/juju/go/src/github.com/juju/juju/api/apiclient.go:1071: unable to connect to API} {/build/juju/parts/juju/go/src/github.com/juju/juju/api/apiclient.go:1096: } {Forbidden}])
10:05:22 DEBUG juju.cmd.juju.commands bootstrap.go:1332 cleaning up after failed bootstrap
10:05:22 INFO  juju.provider.common destroy.go:21 destroying model "controller"
10:05:22 INFO  juju.provider.common destroy.go:32 destroying instances
10:05:24 INFO  juju.provider.common destroy.go:56 destroying storage
10:05:24 INFO  cmd supercommand.go:502 command finished

I have to download things from the net using a proxy, so my config.yaml says:

apt-http-proxy: http://192.168.42.250:3128
apt-https-proxy: http://192.168.42.250:3128
http-proxy: http://192.168.42.250:3128
https-proxy: http://192.168.42.250:3128

Am I missing anything?

2

It seems like you need no-proxy set to exclude the use of the proxy when the connection is to the MAAS nodes themselves. Just a guess.

3

I already added 192.168.42.0/24 to no-proxy but that did not change anything.

4

It seems you have set up the Juju proxy config, but part of the bootstrap process is the Juju CLI connecting to the controller to verify that it is running. So you need to ensure there’s connectivity from your client machine to MAAS as well. This may require no_proxy being set on your client too.
The relevant log lines which show the issue are below.

09:55:24 INFO  cmd controller.go:89 Contacting Juju controller at 192.168.42.101 to verify accessibility...
09:55:24 INFO  juju.juju api.go:67 connecting to API addresses: [192.168.42.101:17070]
10:05:22 ERROR juju.cmd.juju.commands bootstrap.go:697 unable to contact api server after 1 attempts: unable to connect to API: Forbidden
5

I modified my config.yaml like this:

no-proxy: 192.168.42.0/24
apt-http-proxy: http://192.168.42.250:3128
apt-https-proxy: http://192.168.42.250:3128
http-proxy: http://192.168.42.250:3128
https-proxy: http://192.168.42.250:3128

I can curl http://192.168.42.101:17070 -o - which outputs some binary gibberish. When I SSH into that machine and I say netstat -tulpn |grep 17070 I see the service running and listening on all addresses, still it says “Forbidden” after 10 minutes.

Since I wanted to use Juju to install Kubernetes, I think I’ll now resort to manually putting Ubuntu on my nodes with MAAS (this works) and write some Ansible Playbooks to install Kubernetes.

6

Sorry you’ve hit the proxy issues @rabjen-iwes. As @wallyworld was saying the “Forbidden” error is the error proxy servers throw when they’re not happy. The config.yaml it good for the controller, the service running in your cloud. It’ll need to go through proxies to get out to things like streams.canonical.com and retrieve charms from the charmstore.

There’s also communication going on from your laptop/desktop to the controller running on that new VM instance that you bootstrapped to and it seems that’s hitting another proxy. Unfortunately I’m not sure if the local laptop/desktop is setup with the proxy details as the config.yaml is only for the bootstrapped VM side of things.

7

That makes sense. So how do I tell Juju to ignore the proxy when the charm tries to connect to the API address?

8

It’s not the charm, but your laptop running the client software. It’d be interesting to see what your network your laptop is running in and how it connects to this 192.168.42.101 address. I assume your on a different network and there’s a proxy between you and the network your MAAS runs VMs in.

9

I am running this on the MAAS master, and yes, it has the proxy in /etc/environment. Since my work PC is running Windows, I’ll try to set up a HyperV VM running Ubuntu which does not use the proxy, and set up things from there.

10

I just tried it with setting no_proxy to 192.168.42.101,192.168.42.102,192.168.42.103,192.168.42.104,192.168.42.105,192.168.42.106 in both /etc/environment and config.yaml and it did not change anything at all. It still tries connecting for 10 minutes and fails with forbidden.

Guess I’ll resort to installing Kubernetes manually then.