Issue downloading juju agent from controller from MAAS server

juju
,
#1

I have juju installed with MAAS. When I deploy charms from juju cmdline, I see the MAAS grab and assign a machine, I can see the machine gets deployed from the juju status but it never gets out of pending. I have reinstalled/deployed juju a couple of times. When I log into the ‘assigned’ machine I can see it boots and installs with cloud init however the juju agent cant be downloaded. I can download that myself outside of ubuntu machine. not sure why the agent wont download???

Any one have any ideas that can help point me in the right direction to solve this? thanks.

Jun 22 17:31:10 SV5-SU4-FCH1915V1ZL cloud-init[1956]: Attempt 85 to download agent binaries from https://172.30.9.7:17070/model/3b3e07da-eb55-43dc-8029-d27f932840b6/tools/2.9.31-ubuntu-amd64...
Jun 22 17:31:10 SV5-SU4-FCH1915V1ZL cloud-init[1956]: + curl -sSfw agent binaries from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download}
bytes; speed %{speed_download} bytes/s  --connect-timeout 20 --noproxy * --insecure -o /var/lib/juju/tools/2.9.31-ubuntu-amd64/tools.tar.gz https://172.30.9.7:17070/model/3b3e07da-eb55-43dc-8029-d27f932840b6/tools/2.9.31-ubuntu-amd64
Jun 22 17:31:30 SV5-SU4-FCH1915V1ZL cloud-init[1956]: curl: (28) Operation timed out after 20001 milliseconds with 0 out of 0 bytes received
Jun 22 17:31:30 SV5-SU4-FCH1915V1ZL cloud-init[1956]: agent binaries from https://172.30.9.7:17070/model/3b3e07da-eb55-43dc-8029-d27f932840b6/tools/2.9.31-ubuntu-amd64 downloaded: HTTP 000; time 20.001078s; size 0 bytes; speed 0.000 bytes/s + echo Download failed, retrying in 15s
Jun 22 17:31:30 SV5-SU4-FCH1915V1ZL cloud-init[1956]: Download failed, retrying in 15s
Jun 22 17:31:30 SV5-SU4-FCH1915V1ZL cloud-init[1956]: + sleep 15
Jun 22 17:31:45 SV5-SU4-FCH1915V1ZL cloud-init[1956]: + n=86
Jun 22 17:31:45 SV5-SU4-FCH1915V1ZL cloud-init[1956]: + true
Jun 22 17:31:45 SV5-SU4-FCH1915V1ZL cloud-init[1956]: + printf Attempt 86 to download agent binaries from %s...\n https://172.30.9.7:17070/model/3b3e07da-eb55-43dc-8029-d27f932840b6/tools/2.9.31-ubuntu-amd64
Jun 22 17:31:45 SV5-SU4-FCH1915V1ZL cloud-init[1956]: Attempt 86 to download agent binaries from https://172.30.9.7:17070/model/3b3e07da-eb55-43dc-8029-d27f932840b6/tools/2.9.31-ubuntu-amd64...
Jun 22 17:31:45 SV5-SU4-FCH1915V1ZL cloud-init[1956]: + curl -sSfw agent binaries from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download}
bytes; speed %{speed_download} bytes/s  --connect-timeout 20 --noproxy * --insecure -o /var/lib/juju/tools/2.9.31-ubuntu-amd64/tools.tar.gz https://172.30.9.7:17070/model/3b3e07da-eb55-43dc-8029-d27f932840b6/tools/2.9.31-ubuntu-amd64
#2

@mquick thank you for the question! This sounds like a networking issue to me. Are you certain that the networks you’ve configured in MAAS grant access to the outside world? Have you thought through any potential asymmetric routing issues with your networks, and are you familiar with what you’d need to do in order to identify and fix them?

My skills here are a bit too rusty to offer a pat answer in a forum post. But if I were troubleshooting, I’d probably hop onto the box in question and start troubleshooting with tracepath -b and ping.

#3

I can ping the the server fine, they are in the same subnet. we are actually running 1 single flat subnet to simplify networking. If I use the exact URL from a windows machine it will just download the file. the ubuntu OS is 20.04 and curl version is 7.68 I have also tried forcing and upgrade of curl to 7.83.1. Still I am unable to download the file from ubuntu juju controller

#4

I have added the self-signed JUJU-CA to the machine trusted cas trying to download the agent. it seems to be a SSL handshake issue.

using wget

ubuntu@SV5-SU4-FCH1915V1ZL:/usr/local/share/ca-certificates$ wget  -d -v --no-check-certificate  https://172.30.9.7:17070/model/3b3e07da-eb55-43dc-8029-d27f932840b6/tools/2.9.31-ubuntu-amd64
Setting --verbose (verbose) to 1
Setting --verbose (verbose) to 1
Setting --check-certificate (checkcertificate) to 0
Setting --check-certificate (checkcertificate) to 0
DEBUG output created by Wget 1.20.3 on linux-gnu.

Reading HSTS entries from /home/ubuntu/.wget-hsts
URI encoding = ‘UTF-8’
Converted file name '2.9.31-ubuntu-amd64' (UTF-8) -> '2.9.31-ubuntu-amd64' (UTF-8)
--2022-06-22 19:56:02--  https://172.30.9.7:17070/model/3b3e07da-eb55-43dc-8029-d27f932840b6/tools/2.9.31-ubuntu-amd64
Connecting to 172.30.9.7:17070... connected.
Created socket 3.
Releasing 0x0000560b4a6bc0d0 (new refcount 0).
Deleting unused 0x0000560b4a6bc0d0.
Initiating SSL handshake.

using openssl , just sits here, I understand this should show the certchain

OpenSSL> ^C
ubuntu@SV5-SU4-FCH1915V1ZL:/usr/local/share/ca-certificates$ openssl s_client -showcerts -verify 5 -connect 172.30.9.7:17070
verify depth is 5
CONNECTED(00000003)

using curl

ubuntu@SV5-SU4-FCH1915V1ZL:/usr/local/share/ca-certificates$ ^C
ubuntu@SV5-SU4-FCH1915V1ZL:/usr/local/share/ca-certificates$ curl --connect-timeout 20 --noproxy * --insecure -o /var/lib/juju/tools/2.9.31-ubuntu-amd64/tools.tar.gz https://172.30.9.7:17070/model/3b3e07da-eb55-43dc-8029-d27f932840b6/tools/2.9.31-ubuntu-amd64
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:20 --:--:--     0
curl: (28) SSL connection timeout
ubuntu@SV5-SU4-FCH1915V1ZL:/usr/local/share/ca-certificates$

SSL connection timeout??

#6

this shows the local download failing to connect the SSL handshake sequence and in comparison a public site that works. The dashboard server never responds with “Server Hello (2)” I have also upgraded the juju dashboard to 0.9.3 still nothing different

image
image1438×962 214 KB

#7

Thank you for the further information!

Juju is written in Go, and the Go libs can be strict about certificates. I remember running into a similar issue a couple of years ago. The solution was to regenerate the self signed certs with the subject alternative name field filled in. There might be something similar going on here.

#8

ahhh haaa…

well, the problem ended up a being we had a switch with the wrong MTU size. So while we can use telnet and other port testing things, however when you are using a larger packet with TLS the packet was getting trucated.