xe0nic
(Marcin Kubacki)
26 September 2023 14:37
1
Hi,
I’m trying to bootstrap Juju controller using Ubuntu 22.04, and it looks like it has a problem with image metadata record:
juju --debug bootstrap PhysicalOpenStack JujuController \
--config network=5757cb1a-125f-462d-8e7f-97d958db00cf \
--config ssl-hostname-verification=false \
--config default-base=ubuntu@22.04 \
--config default-series=jammy \
--bootstrap-image=9ac1acac-a2bc-4e78-bf4d-925671ecf170 \
--bootstrap-base=ubuntu@22.04 \
--bootstrap-constraints="allocate-public-ip=false arch=amd64" \
--metadata-source /Users/mk/simplestreams
The problem is that it fails:
2023-09-26 13:43:07 DEBUG juju.cmd.jujud bootstrap.go:539 Adding agent binary: 3.2.2-ubuntu-amd64
2023-09-26 13:43:07 DEBUG juju.storage managedstorage.go:239 resource catalog entry created with id "287f022d91b1059cc67b04055d6a9d50b545e511d3c21d1de1b4d9e4b0676c23f4ed1ff0dc22978b1c8903aa60e7101d"
2023-09-26 13:43:07 DEBUG juju.state.pool.txnwatcher txnwatcher.go:487 txn watcher: storedResources 287f022d91b1059cc67b04055d6a9d50b545e511d3c21d1de1b4d9e4b0676c23f4ed1ff0dc22978b1c8903aa60e7101d #2
2023-09-26 13:43:07 DEBUG juju.state.pool.txnwatcher txnwatcher.go:487 txn watcher: storedResources 287f022d91b1059cc67b04055d6a9d50b545e511d3c21d1de1b4d9e4b0676c23f4ed1ff0dc22978b1c8903aa60e7101d #3
2023-09-26 13:43:07 DEBUG juju.storage managedstorage.go:293 managed resource entry created with path "buckets/f7deb1fd-1961-4ea0-80d6-f3281d402f41/tools/3.2.2-ubuntu-amd64-4d64fe6738b6a408b00cdf0b72e642252559b60c988de7af5cac71c397744895" -> "287f022d91b1059cc67b04055d6a9d50b545e511d3c21d1de1b4d9e4b0676c23f4ed1ff0dc22978b1c8903aa60e7101d"
2023-09-26 13:43:07 DEBUG juju.state.pool.txnwatcher txnwatcher.go:487 txn watcher: managedStoredResources buckets/f7deb1fd-1961-4ea0-80d6-f3281d402f41/tools/3.2.2-ubuntu-amd64-4d64fe6738b6a408b00cdf0b72e642252559b60c988de7af5cac71c397744895 #2
2023-09-26 13:43:07 DEBUG juju.state.pool.txnwatcher txnwatcher.go:487 txn watcher: toolsmetadata f7deb1fd-1961-4ea0-80d6-f3281d402f41:3.2.2-ubuntu-amd64 #2
2023-09-26 13:43:07 DEBUG juju.cmd.jujud bootstrap.go:548 saving custom image metadata
2023-09-26 13:43:07 DEBUG juju.state.cloudimagemetadata image.go:87 inserting cloud image metadata for released:RegionOne:22.04:amd64:::custom
2023-09-26 13:43:07 DEBUG juju.state.watcher runner.go:278 killing runner 0xc00013f440
2023-09-26 13:43:07 INFO juju.state.watcher runner.go:386 runner is dying
2023-09-26 13:43:07 DEBUG juju.state.watcher runner.go:535 killing "txnlog"
2023-09-26 13:43:07 INFO juju.state.watcher runner.go:587 stopped "txnlog", err: <nil>
2023-09-26 13:43:07 DEBUG juju.state.watcher runner.go:406 "txnlog" done: <nil>
2023-09-26 13:43:07 DEBUG juju.state.watcher runner.go:473 no restart, removing "txnlog" from known workers
2023-09-26 13:43:07 DEBUG juju.state.pool.txnwatcher runner.go:278 killing runner 0xc00097c120
2023-09-26 13:43:07 INFO juju.state.pool.txnwatcher runner.go:386 runner is dying
2023-09-26 13:43:07 DEBUG juju.state.pool.txnwatcher runner.go:535 killing "txnlog"
2023-09-26 13:43:08 WARNING juju.state.pool.txnwatcher txnwatcher.go:338 txn watcher sync error: tomb: dying
2023-09-26 13:43:08 WARNING juju.state.pool.txnwatcher txnwatcher.go:346 txn watcher resume queued
2023-09-26 13:43:08 INFO juju.state.pool.txnwatcher runner.go:587 stopped "txnlog", err: <nil>
2023-09-26 13:43:08 DEBUG juju.state.pool.txnwatcher runner.go:406 "txnlog" done: <nil>
2023-09-26 13:43:08 DEBUG juju.state.pool.txnwatcher runner.go:473 no restart, removing "txnlog" from known workers
2023-09-26 13:43:08 DEBUG juju.state.watcher runner.go:278 killing runner 0xc00013f440
2023-09-26 13:43:08 DEBUG juju.state open.go:207 closed state without error
ERROR cannot cache image metadata: cannot save cloud image metadata: duplicate metadata record for image id 9ac1acac-a2bc-4e78-bf4d-925671ecf170 (key="released:RegionOne:22.04:amd64:::custom")
2023-09-26 13:43:08 DEBUG cmd supercommand.go:548 error stack:
github.com/juju/juju/state/cloudimagemetadata.(*storage).SaveMetadata.func1:72: duplicate metadata record for image id 9ac1acac-a2bc-4e78-bf4d-925671ecf170 (key="released:RegionOne:22.04:amd64:::custom")
github.com/juju/juju/state/cloudimagemetadata.(*storage).SaveMetadata:107: cannot save cloud image metadata
github.com/juju/juju/cmd/jujud/agent.storeImageMetadataInState:582: cannot cache image metadata
2023-09-26 13:43:08 DEBUG juju.cmd.jujud main.go:289 jujud complete, code 0, err <nil>
15:43:09 ERROR juju.cmd.juju.commands bootstrap.go:984 failed to bootstrap model: subprocess encountered error code 1
15:43:09 DEBUG juju.cmd.juju.commands bootstrap.go:985 (error details: [{github.com/juju/juju/cmd/juju/commands.(*bootstrapCommand).Run:1083: failed to bootstrap model} {github.com/juju/juju/environs/bootstrap.Bootstrap:732: } {github.com/juju/juju/environs/bootstrap.bootstrapIAAS:663: } {subprocess encountered error code 1}])
15:43:09 DEBUG juju.cmd.juju.commands bootstrap.go:1794 cleaning up after failed bootstrap
15:43:09 INFO juju.provider.common destroy.go:21 destroying model "controller"
15:43:09 INFO juju.provider.common destroy.go:32 destroying instances
Got any ideas what could be the problem?
nvinuesa
(Nicolas Vinuesa)
9 October 2023 12:45
2
Hi and welcome @xe0nic !
I’ll try to replicate this on my side, which might take some time since I have to bring up a local openstack environment.
Regarding the image 9ac1acac-a2bc-4e78-bf4d-925671ecf170
, so we can win some time, is it a custom ubuntu image that you have on your simplestreams (/Users/mk/simplestreams
)? I ask because I’ll try to reproduce using a jammy image.
nvinuesa
(Nicolas Vinuesa)
9 October 2023 12:47
3
also, could you please show me the contents of /Users/mk/simplestreams
? specially index.json
and/or index2.json
xe0nic
(Marcin Kubacki)
9 October 2023 13:01
4
➜ ~ cat simplestreams/images/streams/v1/index.json
{
"index": {
"com.ubuntu.cloud:custom": {
"updated": "Tue, 26 Sep 2023 15:06:58 +0200",
"format": "products:1.0",
"datatype": "image-ids",
"cloudname": "custom",
"clouds": [
{
"region": "RegionOne",
"endpoint": "http://10.32.0.2:5000"
}
],
"path": "streams/v1/com.ubuntu.cloud-released-imagemetadata.json",
"products": [
"com.ubuntu.cloud:server:22.04:amd64"
]
}
},
"updated": "Tue, 26 Sep 2023 15:36:58 +0200",
"format": "index:1.0"
}
➜ ~ cat simplestreams/images/streams/v1/com.ubuntu.cloud-released-imagemetadata.json
{
"products": {
"com.ubuntu.cloud:server:22.04:amd64": {
"version": "22.04",
"arch": "amd64",
"versions": {
"20230926": {
"items": {
"9ac1acac-a2bc-4e78-bf4d-925671ecf170": {
"id": "9ac1acac-a2bc-4e78-bf4d-925671ecf170",
"region": "RegionOne",
"endpoint": "http://10.32.0.2:5000"
}
}
}
}
}
},
"updated": "Tue, 26 Sep 2023 15:36:58 +0200",
"format": "products:1.0",
"content_id": "com.ubuntu.cloud:custom"
}
xe0nic
(Marcin Kubacki)
9 October 2023 13:04
5
This is imported Jammy Cloud Image - here is what I did:
[root@c2 ~]# wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
--2023-09-26 15:06:09-- https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
Resolving cloud-images.ubuntu.com (cloud-images.ubuntu.com)... 185.125.190.40, 185.125.190.37, 2620:2d:4000:1::1a, ...
Connecting to cloud-images.ubuntu.com (cloud-images.ubuntu.com)|185.125.190.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 670957568 (640M) [application/octet-stream]
Saving to: 'jammy-server-cloudimg-amd64.img'
jammy-server-cloudimg-amd64.img 100%[======================================================================================================================>] 639.88M 68.4MB/s in 9.6s
2023-09-26 15:06:19 (66.8 MB/s) - 'jammy-server-cloudimg-amd64.img' saved [670957568/670957568]
[root@c2 ~]# openstack image create --disk-format qcow2 --container-format=bare --public --file ./jammy-server-cloudimg-amd64.img -c id jammy
+-------+--------------------------------------+
| Field | Value |
+-------+--------------------------------------+
| id | 9ac1acac-a2bc-4e78-bf4d-925671ecf170 |
+-------+--------------------------------------+
juju metadata generate-image -d ~/simplestreams -i 9ac1acac-a2bc-4e78-bf4d-925671ecf170 --base ubuntu@22.04 -r RegionOne -u http://10.32.0.2:5000
nvinuesa
(Nicolas Vinuesa)
11 October 2023 16:43
6
I haven’t been able to replicate this, I’m waiting for an env. In the meantime, what happens if you don’t provide the --bootstrap-image
flag? I think juju should still pick the correct image from the provided simplestreams since you are providing the bootstrap series.
xe0nic
(Marcin Kubacki)
12 October 2023 10:09
7
Then I have:
12:06:13 DEBUG juju.environs.instances image.go:66 instance constraints {region: RegionOne, base: ubuntu@22.04, arch: amd64, constraints: arch=amd64 mem=3584M allocate-public-ip=false, storage: []}
12:06:13 ERROR juju.cmd.juju.commands bootstrap.go:984 failed to bootstrap model: cannot start bootstrap instance: no metadata for "ubuntu@22.04" images in RegionOne with arch amd64
12:06:13 DEBUG juju.cmd.juju.commands bootstrap.go:985 (error details: [{github.com/juju/juju/cmd/juju/commands.(*bootstrapCommand).Run:1083: failed to bootstrap model} {github.com/juju/juju/environs/bootstrap.Bootstrap:732: } {github.com/juju/juju/environs/bootstrap.bootstrapIAAS:584: } {github.com/juju/juju/provider/common.Bootstrap:61: } {github.com/juju/juju/provider/common.BootstrapInstance:297: cannot start bootstrap instance} {github.com/juju/juju/provider/openstack.(*Environ).StartInstance:1073: } {github.com/juju/juju/environs.ZoneIndependentError:42: no metadata for "ubuntu@22.04" images in RegionOne with arch amd64}])
12:06:13 DEBUG juju.cmd.juju.commands bootstrap.go:1794 cleaning up after failed bootstrap
nvinuesa
(Nicolas Vinuesa)
17 October 2023 15:54
8
ok, I got an o7k env up and running and I can indeed reproduce this. I will look a bit more into detail but probably I will open a bug and keep you informed about it.
nvinuesa
(Nicolas Vinuesa)
18 October 2023 08:03
9
xe0nic
(Marcin Kubacki)
20 November 2023 09:10
10
Hi Nicolas,
any updates when it could be resolved?
nvinuesa
(Nicolas Vinuesa)
21 November 2023 14:43
11
Hi Marcin!
I started looking at this problem more deeply yesterday, but still don’t have a fix on our side. In the meantime could you please try:
juju --debug bootstrap PhysicalOpenStack JujuController \
--config network=5757cb1a-125f-462d-8e7f-97d958db00cf \
--config ssl-hostname-verification=false \
--config default-base=ubuntu@22.04 \
--config default-series=jammy \
--bootstrap-image=9ac1acac-a2bc-4e78-bf4d-925671ecf170 \
--bootstrap-base=ubuntu@22.04 \
--bootstrap-constraints="allocate-public-ip=false arch=amd64" \
If you can see the imageId on the list when openstack image list
then juju should work without --metadata-source
, but I want to see if you get any errors (I tried on my instance and it works).
xe0nic
(Marcin Kubacki)
22 November 2023 08:57
12
It looks like it worked - I have:
Bootstrap complete, controller "jujucontroller" is now available
Thank you