Juju Bootstrap - duplicate metadata record for image id

juju
, , ,
#1

Hi,

I’m trying to bootstrap Juju controller using Ubuntu 22.04, and it looks like it has a problem with image metadata record:

juju --debug bootstrap PhysicalOpenStack JujuController \
--config network=5757cb1a-125f-462d-8e7f-97d958db00cf \
--config ssl-hostname-verification=false \
--config default-base=ubuntu@22.04 \
--config default-series=jammy \
--bootstrap-image=9ac1acac-a2bc-4e78-bf4d-925671ecf170 \
--bootstrap-base=ubuntu@22.04 \
--bootstrap-constraints="allocate-public-ip=false arch=amd64" \
--metadata-source /Users/mk/simplestreams

The problem is that it fails:

2023-09-26 13:43:07 DEBUG juju.cmd.jujud bootstrap.go:539 Adding agent binary: 3.2.2-ubuntu-amd64
2023-09-26 13:43:07 DEBUG juju.storage managedstorage.go:239 resource catalog entry created with id "287f022d91b1059cc67b04055d6a9d50b545e511d3c21d1de1b4d9e4b0676c23f4ed1ff0dc22978b1c8903aa60e7101d"
2023-09-26 13:43:07 DEBUG juju.state.pool.txnwatcher txnwatcher.go:487 txn watcher: storedResources 287f022d91b1059cc67b04055d6a9d50b545e511d3c21d1de1b4d9e4b0676c23f4ed1ff0dc22978b1c8903aa60e7101d #2
2023-09-26 13:43:07 DEBUG juju.state.pool.txnwatcher txnwatcher.go:487 txn watcher: storedResources 287f022d91b1059cc67b04055d6a9d50b545e511d3c21d1de1b4d9e4b0676c23f4ed1ff0dc22978b1c8903aa60e7101d #3
2023-09-26 13:43:07 DEBUG juju.storage managedstorage.go:293 managed resource entry created with path "buckets/f7deb1fd-1961-4ea0-80d6-f3281d402f41/tools/3.2.2-ubuntu-amd64-4d64fe6738b6a408b00cdf0b72e642252559b60c988de7af5cac71c397744895" -> "287f022d91b1059cc67b04055d6a9d50b545e511d3c21d1de1b4d9e4b0676c23f4ed1ff0dc22978b1c8903aa60e7101d"
2023-09-26 13:43:07 DEBUG juju.state.pool.txnwatcher txnwatcher.go:487 txn watcher: managedStoredResources buckets/f7deb1fd-1961-4ea0-80d6-f3281d402f41/tools/3.2.2-ubuntu-amd64-4d64fe6738b6a408b00cdf0b72e642252559b60c988de7af5cac71c397744895 #2
2023-09-26 13:43:07 DEBUG juju.state.pool.txnwatcher txnwatcher.go:487 txn watcher: toolsmetadata f7deb1fd-1961-4ea0-80d6-f3281d402f41:3.2.2-ubuntu-amd64 #2
2023-09-26 13:43:07 DEBUG juju.cmd.jujud bootstrap.go:548 saving custom image metadata
2023-09-26 13:43:07 DEBUG juju.state.cloudimagemetadata image.go:87 inserting cloud image metadata for released:RegionOne:22.04:amd64:::custom
2023-09-26 13:43:07 DEBUG juju.state.watcher runner.go:278 killing runner 0xc00013f440
2023-09-26 13:43:07 INFO juju.state.watcher runner.go:386 runner is dying
2023-09-26 13:43:07 DEBUG juju.state.watcher runner.go:535 killing "txnlog"
2023-09-26 13:43:07 INFO juju.state.watcher runner.go:587 stopped "txnlog", err: <nil>
2023-09-26 13:43:07 DEBUG juju.state.watcher runner.go:406 "txnlog" done: <nil>
2023-09-26 13:43:07 DEBUG juju.state.watcher runner.go:473 no restart, removing "txnlog" from known workers
2023-09-26 13:43:07 DEBUG juju.state.pool.txnwatcher runner.go:278 killing runner 0xc00097c120
2023-09-26 13:43:07 INFO juju.state.pool.txnwatcher runner.go:386 runner is dying
2023-09-26 13:43:07 DEBUG juju.state.pool.txnwatcher runner.go:535 killing "txnlog"
2023-09-26 13:43:08 WARNING juju.state.pool.txnwatcher txnwatcher.go:338 txn watcher sync error: tomb: dying
2023-09-26 13:43:08 WARNING juju.state.pool.txnwatcher txnwatcher.go:346 txn watcher resume queued
2023-09-26 13:43:08 INFO juju.state.pool.txnwatcher runner.go:587 stopped "txnlog", err: <nil>
2023-09-26 13:43:08 DEBUG juju.state.pool.txnwatcher runner.go:406 "txnlog" done: <nil>
2023-09-26 13:43:08 DEBUG juju.state.pool.txnwatcher runner.go:473 no restart, removing "txnlog" from known workers
2023-09-26 13:43:08 DEBUG juju.state.watcher runner.go:278 killing runner 0xc00013f440
2023-09-26 13:43:08 DEBUG juju.state open.go:207 closed state without error
ERROR cannot cache image metadata: cannot save cloud image metadata: duplicate metadata record for image id 9ac1acac-a2bc-4e78-bf4d-925671ecf170 (key="released:RegionOne:22.04:amd64:::custom")
2023-09-26 13:43:08 DEBUG cmd supercommand.go:548 error stack:
github.com/juju/juju/state/cloudimagemetadata.(*storage).SaveMetadata.func1:72: duplicate metadata record for image id 9ac1acac-a2bc-4e78-bf4d-925671ecf170 (key="released:RegionOne:22.04:amd64:::custom")
github.com/juju/juju/state/cloudimagemetadata.(*storage).SaveMetadata:107: cannot save cloud image metadata
github.com/juju/juju/cmd/jujud/agent.storeImageMetadataInState:582: cannot cache image metadata
2023-09-26 13:43:08 DEBUG juju.cmd.jujud main.go:289 jujud complete, code 0, err <nil>
15:43:09 ERROR juju.cmd.juju.commands bootstrap.go:984 failed to bootstrap model: subprocess encountered error code 1
15:43:09 DEBUG juju.cmd.juju.commands bootstrap.go:985 (error details: [{github.com/juju/juju/cmd/juju/commands.(*bootstrapCommand).Run:1083: failed to bootstrap model} {github.com/juju/juju/environs/bootstrap.Bootstrap:732: } {github.com/juju/juju/environs/bootstrap.bootstrapIAAS:663: } {subprocess encountered error code 1}])
15:43:09 DEBUG juju.cmd.juju.commands bootstrap.go:1794 cleaning up after failed bootstrap
15:43:09 INFO juju.provider.common destroy.go:21 destroying model "controller"
15:43:09 INFO juju.provider.common destroy.go:32 destroying instances

Got any ideas what could be the problem?

#2

Hi and welcome @xe0nic!

I’ll try to replicate this on my side, which might take some time since I have to bring up a local openstack environment.

Regarding the image 9ac1acac-a2bc-4e78-bf4d-925671ecf170, so we can win some time, is it a custom ubuntu image that you have on your simplestreams (/Users/mk/simplestreams)? I ask because I’ll try to reproduce using a jammy image.

#3

also, could you please show me the contents of /Users/mk/simplestreams? specially index.json and/or index2.json

#4 
➜  ~ cat simplestreams/images/streams/v1/index.json 
{
    "index": {
        "com.ubuntu.cloud:custom": {
            "updated": "Tue, 26 Sep 2023 15:06:58 +0200",
            "format": "products:1.0",
            "datatype": "image-ids",
            "cloudname": "custom",
            "clouds": [
                {
                    "region": "RegionOne",
                    "endpoint": "http://10.32.0.2:5000"
                }
            ],
            "path": "streams/v1/com.ubuntu.cloud-released-imagemetadata.json",
            "products": [
                "com.ubuntu.cloud:server:22.04:amd64"
            ]
        }
    },
    "updated": "Tue, 26 Sep 2023 15:36:58 +0200",
    "format": "index:1.0"
}
➜  ~ cat simplestreams/images/streams/v1/com.ubuntu.cloud-released-imagemetadata.json 
{
    "products": {
        "com.ubuntu.cloud:server:22.04:amd64": {
            "version": "22.04",
            "arch": "amd64",
            "versions": {
                "20230926": {
                    "items": {
                        "9ac1acac-a2bc-4e78-bf4d-925671ecf170": {
                            "id": "9ac1acac-a2bc-4e78-bf4d-925671ecf170",
                            "region": "RegionOne",
                            "endpoint": "http://10.32.0.2:5000"
                        }
                    }
                }
            }
        }
    },
    "updated": "Tue, 26 Sep 2023 15:36:58 +0200",
    "format": "products:1.0",
    "content_id": "com.ubuntu.cloud:custom"
}
#5

This is imported Jammy Cloud Image - here is what I did:

[root@c2 ~]# wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
--2023-09-26 15:06:09--  https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
Resolving cloud-images.ubuntu.com (cloud-images.ubuntu.com)... 185.125.190.40, 185.125.190.37, 2620:2d:4000:1::1a, ...
Connecting to cloud-images.ubuntu.com (cloud-images.ubuntu.com)|185.125.190.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 670957568 (640M) [application/octet-stream]
Saving to: 'jammy-server-cloudimg-amd64.img'

jammy-server-cloudimg-amd64.img                      100%[======================================================================================================================>] 639.88M  68.4MB/s    in 9.6s    

2023-09-26 15:06:19 (66.8 MB/s) - 'jammy-server-cloudimg-amd64.img' saved [670957568/670957568]

[root@c2 ~]# openstack image create --disk-format qcow2 --container-format=bare --public --file ./jammy-server-cloudimg-amd64.img -c id jammy
+-------+--------------------------------------+
| Field | Value                                |
+-------+--------------------------------------+
| id    | 9ac1acac-a2bc-4e78-bf4d-925671ecf170 |
+-------+--------------------------------------+

juju metadata generate-image -d ~/simplestreams -i 9ac1acac-a2bc-4e78-bf4d-925671ecf170 --base ubuntu@22.04 -r RegionOne -u http://10.32.0.2:5000
#6

I haven’t been able to replicate this, I’m waiting for an env. In the meantime, what happens if you don’t provide the --bootstrap-image flag? I think juju should still pick the correct image from the provided simplestreams since you are providing the bootstrap series.

#7

Then I have:

12:06:13 DEBUG juju.environs.instances image.go:66 instance constraints {region: RegionOne, base: ubuntu@22.04, arch: amd64, constraints: arch=amd64 mem=3584M allocate-public-ip=false, storage: []}
12:06:13 ERROR juju.cmd.juju.commands bootstrap.go:984 failed to bootstrap model: cannot start bootstrap instance: no metadata for "ubuntu@22.04" images in RegionOne with arch amd64
12:06:13 DEBUG juju.cmd.juju.commands bootstrap.go:985 (error details: [{github.com/juju/juju/cmd/juju/commands.(*bootstrapCommand).Run:1083: failed to bootstrap model} {github.com/juju/juju/environs/bootstrap.Bootstrap:732: } {github.com/juju/juju/environs/bootstrap.bootstrapIAAS:584: } {github.com/juju/juju/provider/common.Bootstrap:61: } {github.com/juju/juju/provider/common.BootstrapInstance:297: cannot start bootstrap instance} {github.com/juju/juju/provider/openstack.(*Environ).StartInstance:1073: } {github.com/juju/juju/environs.ZoneIndependentError:42: no metadata for "ubuntu@22.04" images in RegionOne with arch amd64}])
12:06:13 DEBUG juju.cmd.juju.commands bootstrap.go:1794 cleaning up after failed bootstrap
#8

ok, I got an o7k env up and running and I can indeed reproduce this. I will look a bit more into detail but probably I will open a bug and keep you informed about it.

#9

Bug #2039648 “Error bootstrapping on openstack when specifying b...” : Bugs : Canonical Juju here it is @xe0nic

I’ll try to get on it as soon as possible.

#10

Hi Nicolas,

any updates when it could be resolved?

#11

Hi Marcin!

I started looking at this problem more deeply yesterday, but still don’t have a fix on our side. In the meantime could you please try:

juju --debug bootstrap PhysicalOpenStack JujuController \
--config network=5757cb1a-125f-462d-8e7f-97d958db00cf \
--config ssl-hostname-verification=false \
--config default-base=ubuntu@22.04 \
--config default-series=jammy \
--bootstrap-image=9ac1acac-a2bc-4e78-bf4d-925671ecf170 \
--bootstrap-base=ubuntu@22.04 \
--bootstrap-constraints="allocate-public-ip=false arch=amd64" \

If you can see the imageId on the list when openstack image list then juju should work without --metadata-source, but I want to see if you get any errors (I tried on my instance and it works).

#12

It looks like it worked - I have:

Bootstrap complete, controller "jujucontroller" is now available

Thank you