Charmed openstack hook failed config changed

charm
1

Hello guys,

I’m trying to deploy a charmed Openstack cloud but haven’t had success with it. The bundle I’m using to deploy openstack is this one

However, the OSD Ceph nodes and the compute nodes keep on failing with the message hook failed: “config-changed”

From what I can see from the logs, I has something to do with the repositories not being valid:

Nova compute logs

2022-09-21 17:46:01 WARNING unit.nova-compute/0.juju-log server.go:316 Package openstack-release has no installation candidate.
2022-09-21 17:46:02 INFO unit.nova-compute/0.juju-log server.go:316 Performing OpenStack upgrade to yoga.
2022-09-21 17:46:02 INFO unit.nova-compute/0.juju-log server.go:316 DEPRECATION WARNING: Function configure_installation_source is being removed on/around 2017-07 : use charmhelpers.fetch.add_source() instead.
2022-09-21 17:46:02 INFO unit.nova-compute/0.juju-log server.go:316 Installing [] with options: ['--option=Dpkg::Options::=--force-confold']
2022-09-21 17:46:06 WARNING unit.nova-compute/0.config-changed logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
2022-09-21 17:46:06 WARNING unit.nova-compute/0.config-changed logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.
2022-09-21 17:46:06 INFO unit.nova-compute/0.juju-log server.go:316 Couldn't acquire DPKG lock. Will retry in 10 seconds
2022-09-21 17:46:21 WARNING unit.nova-compute/0.config-changed logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
2022-09-21 17:46:21 WARNING unit.nova-compute/0.config-changed logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.
2022-09-21 17:46:21 INFO unit.nova-compute/0.juju-log server.go:316 Couldn't acquire DPKG lock. Will retry in 10 seconds
2022-09-21 17:46:33 WARNING unit.nova-compute/0.config-changed logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
2022-09-21 17:46:33 WARNING unit.nova-compute/0.config-changed logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.
2022-09-21 17:46:33 INFO unit.nova-compute/0.juju-log server.go:316 Couldn't acquire DPKG lock. Will retry in 10 seconds
2022-09-21 17:46:45 WARNING unit.nova-compute/0.config-changed logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
2022-09-21 17:46:45 WARNING unit.nova-compute/0.config-changed logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.

Ceph OSD Logs

> 022-09-21 17:44:40 WARNING unit.ceph-osd/1.install logger.go:60   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
> 2022-09-21 17:44:40 WARNING unit.ceph-osd/1.install logger.go:60     raise CalledProcessError(retcode, cmd)
> 2022-09-21 17:44:40 WARNING unit.ceph-osd/1.install logger.go:60 subprocess.CalledProcessError: Command '['apt-get', 'update']' returned non-zero exit status 100.
> 2022-09-21 17:44:40 ERROR juju.worker.uniter.operation runhook.go:140 hook "install" (via explicit, bespoke hook script) failed: exit status 1
> 2022-09-21 17:44:40 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "install" hook
> 2022-09-21 17:47:31 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "install" hook
> 2022-09-21 17:49:40 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "install" hook
> 2022-09-21 17:49:41 INFO unit.ceph-osd/1.juju-log server.go:316 Installing [] with options: ['--option=Dpkg::Options::=--force-confold']
> 2022-09-21 17:49:45 WARNING unit.ceph-osd/1.install logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
> 2022-09-21 17:49:45 WARNING unit.ceph-osd/1.install logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.
> 2022-09-21 17:49:45 INFO unit.ceph-osd/1.juju-log server.go:316 Couldn't acquire DPKG lock. Will retry in 10 seconds
> 2022-09-21 17:49:59 WARNING unit.ceph-osd/1.install logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
> 2022-09-21 17:49:59 WARNING unit.ceph-osd/1.install logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.
> 2022-09-21 17:49:59 INFO unit.ceph-osd/1.juju-log server.go:316 Couldn't acquire DPKG lock. Will retry in 10 seconds
> 2022-09-21 17:50:13 WARNING unit.ceph-osd/1.install logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
> 2022-09-21 17:50:13 WARNING unit.ceph-osd/1.install logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.
> 2022-09-21 17:50:13 INFO unit.ceph-osd/1.juju-log server.go:316 Couldn't acquire DPKG lock. Will retry in 10 seconds
> 2022-09-21 17:50:25 WARNING unit.ceph-osd/1.install logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
> 2022-09-21 17:50:25 WARNING unit.ceph-osd/1.install logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.
> 2022-09-21 17:50:25 INFO unit.ceph-osd/1.juju-log server.go:316 Couldn't acquire DPKG lock. Will retry in 10 seconds
> 2022-09-21 17:50:38 WARNING unit.ceph-osd/1.install logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
> 2022-09-21 17:50:38 WARNING unit.ceph-osd/1.install logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.
> 2022-09-21 17:50:38 INFO unit.ceph-osd/1.juju-log server.go:316 Couldn't acquire DPKG lock. Will retry in 10 seconds
> 2022-09-21 17:50:50 WARNING unit.ceph-osd/1.install logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
> 2022-09-21 17:50:50 WARNING unit.ceph-osd/1.install logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.
> 2022-09-21 17:50:50 INFO unit.ceph-osd/1.juju-log server.go:316 Couldn't acquire DPKG lock. Will retry in 10 seconds
> 2022-09-21 17:51:02 WARNING unit.ceph-osd/1.install logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
> 2022-09-21 17:51:02 WARNING unit.ceph-osd/1.install logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.
> 2022-09-21 17:51:02 INFO unit.ceph-osd/1.juju-log server.go:316 Couldn't acquire DPKG lock. Will retry in 10 seconds
> 2022-09-21 17:51:15 WARNING unit.ceph-osd/1.install logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
> 2022-09-21 17:51:15 WARNING unit.ceph-osd/1.install logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.
> 2022-09-21 17:51:15 INFO unit.ceph-osd/1.juju-log server.go:316 Couldn't acquire DPKG lock. Will retry in 10 seconds
> 2022-09-21 17:51:28 WARNING unit.ceph-osd/1.install logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
> 2022-09-21 17:51:28 WARNING unit.ceph-osd/1.install logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.
> 2022-09-21 17:51:28 INFO unit.ceph-osd/1.juju-log server.go:316 Couldn't acquire DPKG lock. Will retry in 10 seconds
> 2022-09-21 17:51:40 WARNING unit.ceph-osd/1.install logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
> 2022-09-21 17:51:40 WARNING unit.ceph-osd/1.install logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.
> 2022-09-21 17:51:40 INFO unit.ceph-osd/1.juju-log server.go:316 Couldn't acquire DPKG lock. Will retry in 10 seconds
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60 Traceback (most recent call last):
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60   File "/var/lib/juju/agents/unit-ceph-osd-1/charm/hooks/install.real", line 911, in <module>
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60     hooks.execute(sys.argv)
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60   File "/var/lib/juju/agents/unit-ceph-osd-1/charm/hooks/charmhelpers/core/hookenv.py", line 963, in execute
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60     self._hooks[hook_name]()
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60   File "/var/lib/juju/agents/unit-ceph-osd-1/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 90, in _harden_inner2
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60     return f(*args, **kwargs)
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60   File "/var/lib/juju/agents/unit-ceph-osd-1/charm/hooks/install.real", line 320, in install
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60     apt_update(fatal=True)
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60   File "/var/lib/juju/agents/unit-ceph-osd-1/charm/hooks/charmhelpers/fetch/ubuntu.py", line 400, in apt_update
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60     _run_apt_command(cmd, fatal)
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60   File "/var/lib/juju/agents/unit-ceph-osd-1/charm/hooks/charmhelpers/fetch/ubuntu.py", line 945, in _run_apt_command
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60     _run_with_retries(
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60   File "/var/lib/juju/agents/unit-ceph-osd-1/charm/hooks/charmhelpers/fetch/ubuntu.py", line 922, in _run_with_retries
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60     result = subprocess.check_call(cmd, env=env, **kwargs)
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60     raise CalledProcessError(retcode, cmd)
> 2022-09-21 17:51:53 WARNING unit.ceph-osd/1.install logger.go:60 subprocess.CalledProcessError: Command '['apt-get', 'update']' returned non-zero exit status 100.
> 2022-09-21 17:51:53 ERROR juju.worker.uniter.operation runhook.go:140 hook "install" (via explicit, bespoke hook script) failed: exit status 1
> 2022-09-21 17:51:53 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "install" hook

I’m trying to use Focal 20.04 and Openstack Yoga with no luck. I’ve tried to use Jammy 22.04 but I get an error from a few charms saying that jammy is not supported (i.e. Barbican).

Do you have any ideas why I’m getting this error?

Thank you in advance.

2

This usually is an indication that apt processes are currently already running. Check the running processes on the affected machines.

That being said, to use Yoga on Focal I recommend looking at the main Charmed OpenStack tutorial. It uses charm channels. You seem to be using legacy non-channel charms.

As for the barbican charm not being supported on Jammy:

$ juju add-model --config default-series=jammy barbican
Added 'barbican' model on mystack/mystack with credential 'pmatulis' for user 'admin'

$ juju deploy --channel yoga/stable barbican
Located charm "barbican" in charm-hub, revision 72
Deploying "barbican" from charm-hub charm "barbican", revision 72 in channel yoga/stable on jammy
3

Hi,

I’ve made a lot of changes to my bundle script following your recomendations, and this is the resulting file.

However, I realized I kept on getting an error about the repositories not being signed only on CEPH OSD nodes and Nova Compute Nodes.

2022-09-22 16:30:33 WARNING unit.nova-compute/0.config-changed logger.go:60 E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/InRelease  403  Forbidden [IP: 172.16.55.91 8000]
2022-09-21 16:30:33 WARNING unit.nova-compute/0.config-changed logger.go:60 E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/yoga InRelease' is not signed.

I realized that if I delete this file:

/etc/apt/sources.list.d/cloud-archive.list

Everything now runs but then I get an error about incompatibilities between the nova controller and the nova compute versions being different. I assume that without the cloud-archive the node just goes to the latest version and that’s why I’m getting the version missmatch.

Do you know why I could be getting the certificate not signed only on those two unit types?

Thank you once again.

4

Is it possible that there is a proxy/firewall on your end that is preventing the plagued machines from accessing the UCA (ubuntu-cloud-archive)?

5

I’ve forced an update on the nodes and noticed this:

E: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-updates/yoga/main/binary-amd64/Packages  403  Forbidden [IP: 172.16.55.91 8000]
E: Some index files failed to download. They have been ignored, or old ones used instead.

The forbidden IP Forbidden [IP: 172.16.55.91 8000] is the MAAS Server IP used for PXE Boot. That IP doesn’t have a default gateway, since I didn’t think one was needed. Also, the error happens only on CEPH OSD Nodes and Compute nodes, which are bare metal nodes. I don’t get an error when deploying anything to LXD Containers for the rest of the Openstack services. Also, after removing that specific configuration file, apt updates and upgrades the server without issue. It is something related to only that file or that repository on either bare metal nodes or CEPH / Nova-compute services.

6

So, I’ve set on maas not to use a proxy and everything seems to be working so far. Will update once the cloud deployment is complete.

7

It sounds like you are making progress. Well done. As this does appear to be a MAAS issue you may get better support on the MAAS user forum.