Juju Controller Bootstrap Issue

#1

Hello ,

I am facing issue while bootstrapping Juju Controller.
Juju Controller is currently a VM on MAAS KVM POD. Juju Bootstrap process gets stucked at “Running machine configuration script…”

Have configured NATing and Routing as below.

sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE  
sudo iptables -A FORWARD -i eth0 -o br1 -m state \
    --state RELATED,ESTABLISHED -j ACCEPT  
sudo iptables -A FORWARD -i br1 -o eth0 -j ACCEPT  
echo 'net.ipv4.ip_forward=1' | sudo tee -a /etc/sysctl.conf  
sudo sysctl -p

juju bootstrap --bootstrap-series=focal --constraints tags=juju nxtgen-poc nxtgen-poc-controller
Creating Juju controller “nxtgen-poc-controller” on nxtgen-poc/default
Looking for packaged Juju agent version 2.8.9 for amd64
Launching controller instance(s) on nxtgen-poc/default…

  • empwes (arch=amd64 mem=4G cores=2)
    Installing Juju agent on bootstrap instance
    Fetching Juju Dashboard 0.3.0
    Waiting for address
    Attempting to connect to 192.168.30.3:22
    Connected to 192.168.30.3
    Running machine configuration script…

/var/log/cloud-init.log

2021-03-21 19:34:08,272 - util.py[DEBUG]: Creating symbolic link from
‘/run/cloud-init/result.json’ =>
‘…/…/var/lib/cloud/data/result.json’ 2021-03-21 19:34:08,272 -
util.py[DEBUG]: Reading from /proc/uptime (quiet=False) 2021-03-21
19:34:08,273 - util.py[DEBUG]: Read 12 bytes from /proc/uptime
2021-03-21 19:34:08,273 - util.py[DEBUG]: cloud-init mode ‘modules’
took 10.367 seconds (10.37) 2021-03-21 19:34:08,273 -
handlers.py[DEBUG]: finish: modules-final: SUCCESS: running modules
for final 2021-03-21 19:34:08,273 - url_helper.py[DEBUG]: [0/1] open
‘http://192-168-30-0--24.maas-internal:5248/MAAS/metadata/status/empwes’
with {‘url’:
‘http://192-168-30-0--24.maas-internal:5248/MAAS/metadata/status/empwes’,
‘allow_redirects’: True, ‘method’: ‘POST’, ‘headers’:
{‘Authorization’: ‘OAuth oauth_nonce=“32825240929956351451616355248”,
oauth_timestamp=“1616355248”, oauth_version=“1.0”,
oauth_signature_method=“PLAINTEXT”,
oauth_consumer_key=“YgQFkxTVeyzVFqsUyV”,
oauth_token=“jpFZfGNB7h45bMQP8P”,
oauth_signature="%26VYsvZ78qgC86KKLKm7vrUtCwnrQsSbHM"’}} configuration
2021-03-21 19:34:08,291 - url_helper.py[DEBUG]: Read from
http://192-168-30-0--24.maas-internal:5248/MAAS/metadata/status/empwes
(204, 0b) after 1 attempts

Please help to solve this issue.

#2

This looks like it is failing to talk to the maas api to get the metadata for the machine that is starting up.
Is it possible for you to try provisioning a machine against maas without Juju in the mix, and make sure that piece is working?

2 Likes
#3

This might be better discussed

Since it seems to be an issue with the configuration of your MAAS cloud.

1 Like
#4

Seems like we should harden all the places where Juju depends on a cloud service, such as this, and provide much clearer failure messages. IF this said:

jujud trying to connect to MAAS metadata service on 192.168.45.32:33567…
juju failed to fetch MAAS metadata, please check networking between node IP and MAAS API
Failed to bootstrap Juju server.

… then it would be a lot easier for someone to self-serve.

#5

Have isolated issue at Juju Controller VM level. This VM was not having access to outbound internet.
Seems some part of machine configuration script requires outbound internet access at Juju Controller level.

#6

So I believe this is actually before we get to any pieces of Juju, where cloud-init itself is trying to contact the metadata server to find out how the instance itself is going to be configured. (eg, it is happening before we have even told it what to do.)

That said, I might be reading the log incorrectly, and this step succeeded, and it is a later step that failed.

#7

I do think there are some stages like updating apt repositories/installing dependencies that should be possible to redirect to a local mirror via bootstrap config (apt-mirror, apt-http-proxy, etc). If you are bootstrapping onto focal, though, we will be installing our database via snaps. You should also be able to set up a snap store proxy, if you want that to also only be a local connection.