sunbeam fails to download sunbeam-machine charm

Hello,

So i followed the official 1 VM setup for sunbeam with microstack from here: Microstack , excluding the multipass Vm creation, as I deployed one VM with 16cpu and 32gb ram on my charmed openstack setup which is functional.

Ran:

sunbeam prepare-node-script | bash -x && newgrp snap_daemon

All ok, finished correctly, then the next step was to run

sunbeam cluster bootstrap --accept-defaults

This took a long time and now is stuck with this state:

Adding Sunbeam Machine unit to machine sunbeam.xxx

I checked the unit logs and saw this:

2023-11-21 08:53:54 INFO juju.worker.uniter uniter.go:338 unit "sunbeam-machine/0" shutting down: preparing operation "install ch:amd64/jammy/sunbeam-machine-1" for sunbeam-machine/0: failed to download charm "ch:amd64/jammy/sunbeam-machine-1" from API server: download request with archiveSha256 length 0 not valid
2023-11-21 08:53:54 ERROR juju.worker.dependency engine.go:695 "uniter" manifold worker returned unexpected error: preparing operation "install ch:amd64/jammy/sunbeam-machine-1" for sunbeam-machine/0: failed to download charm "ch:amd64/jammy/sunbeam-machine-1" from API server: download request with archiveSha256 length 0 not valid
2023-11-21 08:54:02 INFO juju.worker.uniter uniter.go:354 unit "sunbeam-machine/0" started
2023-11-21 08:54:02 INFO juju.worker.uniter uniter.go:672 resuming charm install
2023-11-21 08:54:02 INFO juju.worker.uniter.charm bundles.go:82 downloading ch:amd64/jammy/sunbeam-machine-1 from API server

It’s doing that for ~30 minutes and the exits with timeout.

juju status

ubuntu@sunbeam:/var/log/juju$ juju status
Model       Controller          Cloud/Region     Version  SLA          Timestamp  Notes
controller  sunbeam-controller  sunbeam/default  3.2.0    unsupported  09:18:21Z  upgrade available: 3.2.3

App              Version  Status   Scale  Charm            Channel      Rev  Exposed  Message
controller                active       1  juju-controller  3.2/stable    14  no
sunbeam-machine           waiting    0/1  sunbeam-machine  latest/edge    1  no       agent initialising

Unit                Workload  Agent       Machine  Public address  Ports  Message
controller/0*       active    idle        0        10.99.0.7
sunbeam-machine/1*  waiting   allocating  0        10.99.0.7              agent initialising

Machine  State    Address    Inst id  Base          AZ  Message
0        started  10.99.0.7  manual:  ubuntu@22.04      Manually provisioned machine

juju debug-log

machine-0: 09:15:13 INFO juju.downloader downloading from ch:amd64/jammy/sunbeam-machine-1
unit-sunbeam-machine-1: 09:15:13 INFO juju.worker.uniter unit "sunbeam-machine/1" shutting down: preparing operation "install ch:amd64/jammy/sunbeam-machine-1" for sunbeam-machine/1: failed to download charm "ch:amd64/jammy/sunbeam-machine-1" from API server: download request with archiveSha256 length 0 not valid
unit-sunbeam-machine-1: 09:15:13 ERROR juju.worker.dependency "uniter" manifold worker returned unexpected error: preparing operation "install ch:amd64/jammy/sunbeam-machine-1" for sunbeam-machine/1: failed to download charm "ch:amd64/jammy/sunbeam-machine-1" from API server: download request with archiveSha256 length 0 not valid
machine-0: 09:15:18 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-sunbeam-machine-1
machine-0: 09:15:18 INFO juju.agent.tools was a symlink, now looking at /var/lib/juju/tools/3.2.0-ubuntu-amd64
unit-sunbeam-machine-1: 09:15:18 INFO juju.worker.uniter unit "sunbeam-machine/1" started
unit-sunbeam-machine-1: 09:15:18 INFO juju.worker.uniter resuming charm install
machine-0: 09:15:18 INFO juju.downloader downloading from ch:amd64/jammy/sunbeam-machine-1
unit-sunbeam-machine-1: 09:15:18 INFO juju.worker.uniter.charm downloading ch:amd64/jammy/sunbeam-machine-1 from API server
unit-sunbeam-machine-1: 09:15:18 INFO juju.worker.uniter unit "sunbeam-machine/1" shutting down: preparing operation "install ch:amd64/jammy/sunbeam-machine-1" for sunbeam-machine/1: failed to download charm "ch:amd64/jammy/sunbeam-machine-1" from API server: download request with archiveSha256 length 0 not valid
unit-sunbeam-machine-1: 09:15:18 ERROR juju.worker.dependency "uniter" manifold worker returned unexpected error: preparing operation "install ch:amd64/jammy/sunbeam-machine-1" for sunbeam-machine/1: failed to download charm "ch:amd64/jammy/sunbeam-machine-1" from API server: download request with archiveSha256 length 0 not valid
machine-0: 09:15:23 INFO juju.worker.leaseexpiry expired 1 leases
machine-0: 09:15:24 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-sunbeam-machine-1
machine-0: 09:15:24 INFO juju.agent.tools was a symlink, now looking at /var/lib/juju/tools/3.2.0-ubuntu-amd64
unit-sunbeam-machine-1: 09:15:24 INFO juju.worker.uniter unit "sunbeam-machine/1" started
unit-sunbeam-machine-1: 09:15:24 INFO juju.worker.uniter resuming charm install
unit-sunbeam-machine-1: 09:15:24 INFO juju.worker.uniter.charm downloading ch:amd64/jammy/sunbeam-machine-1 from API server
machine-0: 09:15:24 INFO juju.downloader downloading from ch:amd64/jammy/sunbeam-machine-1
unit-sunbeam-machine-1: 09:15:24 INFO juju.worker.uniter unit "sunbeam-machine/1" shutting down: preparing operation "install ch:amd64/jammy/sunbeam-machine-1" for sunbeam-machine/1: failed to download charm "ch:amd64/jammy/sunbeam-machine-1" from API server: download request with archiveSha256 length 0 not valid
unit-sunbeam-machine-1: 09:15:24 ERROR juju.worker.dependency "uniter" manifold worker returned unexpected error: preparing operation "install ch:amd64/jammy/sunbeam-machine-1" for sunbeam-machine/1: failed to download charm "ch:amd64/jammy/sunbeam-machine-1" from API server: download request with archiveSha256 length 0 not valid
machine-0: 09:18:37 INFO juju.state LogTailer starting oplog tailing: recent id count=10, lastTime=2023-11-21 09:15:59.596528697 +0000 UTC, minOplogTs=2023-11-21 09:14:59.596528697 +0000 UTC

VM has connectivity to Internet, no MTU problems, nothing like that.

Any thoughts are appreciated.

Thanks

Seems like this might be related to:

however that might just be the same symptom rather than the cause; this is the first charm that gets downloaded as part of the bootstrap process which smells like an issue contacting the charmhub.