Im using:
- Channel: Victoria
- OS: focal
Recently my cluster got power cut. Then I had to redeploy juju apps nova-compute, keystone, neutron-api, vault,nova-cloud-controller…
Now it got stuck at nova-cloud-controller, at first, it got error because there was no file db-client.ca (even no ssl config specified in nova.cond). Then I had to manually create db-client.ca, it seems migrations was successfully, then I test some functions, then I cannot create snapshot, and creating new instance stuck at Scheduling state. From nova-cloud-controller host, I got this error log:
2025-02-17 07:57:26.960 1516 INFO nova.compute.rpcapi [req-b9ed4975-6d8f-44d8-87bc-863055657831 - - - - -] Automatically selected compute RPC version 5.12 from minimum service version 53 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server [req-b9ed4975-6d8f-44d8-87bc-863055657831 - - - - -] Exception during message handling: amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile. 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server
File “/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py”, line 165, in _process_incoming 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py”, line 309, in dispatch 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py”, line 229, in _do_dispatch 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/nova/conductor/manager.py”, line 1686, in schedule_and_build_instances 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server
self.compute_rpcapi.build_and_run_instance( 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/nova/compute/rpcapi.py”, line 1460, in build_and_run_instance 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server cctxt.cast(ctxt, ‘build_and_run_instance’, **kwargs) 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py”, line 150, in cast 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server self.transport._send(self.target, msg_ctxt, msg, 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/oslo_messaging/transport.py”, line 123, in _send 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server return self._driver.send(target, ctxt, message, 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py”, line 680, in send 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server return self._send(target, ctxt, message, wait_for_reply, timeout, 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py”, line 644, in _send 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server with self._get_connection(rpc_common.PURPOSE_SEND) as conn: 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py”, line 597, in _get_connection 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server return rpc_common.ConnectionContext(self._connection_pool, 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/oslo_messaging/_drivers/common.py”, line 423, in init 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server self.connection = connection_pool.get() 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/oslo_messaging/_drivers/pool.py”, line 109, in get 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server return self.create() 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/oslo_messaging/_drivers/pool.py”, line 146, in create 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server return self.connection_cls(self.conf, self.url, purpose) 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py”, line 625, in init 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server self.ensure_connection() 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py”, line 742, in ensure_connection 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server
self.connection.ensure_connection(errback=on_error) 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/kombu/connection.py”, line 389, in ensure_connection 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server self._ensure_connection(*args, **kwargs) 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/kombu/connection.py”, line 441, in _ensure_connection 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server return retry_over_time( 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/kombu/utils/functional.py”, line 344, in retry_over_time 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server return fun(*args, **kwargs) 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/kombu/connection.py”, line 874, in _connection_factory 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server self._connection = self._establish_connection() 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/kombu/connection.py”, line 809, in _establish_connection 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server conn = self.transport.establish_connection() 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/kombu/transport/pyamqp.py”, line 130, in establish_connection 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server conn.connect() 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/amqp/connection.py”, line 320, in connect 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server
self.drain_events(timeout=self.connect_timeout) 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/amqp/connection.py”, line 508, in drain_events 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server while not self.blocking_read(timeout): 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/amqp/connection.py”, line 514, in blocking_read 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server return self.on_inbound_frame(frame) 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/amqp/method_framing.py”, line 55, in on_frame 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server callback(channel, method_sig, buf, None) 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/amqp/connection.py”, line 520, in on_inbound_method 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server return self.channels[channel_id].dispatch_method( 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/amqp/abstract_channel.py”, line 145, in dispatch_method 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server listener(*args) 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server File “/usr/lib/python3/dist-packages/amqp/connection.py”, line 650, in _on_close 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server raise error_for_code(reply_code, reply_text, 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile. 2025-02-17 07:57:26.976 1516 ERROR oslo_messaging.rpc.server
From rabbitmq, there is no log indicates about connection error.
juju status: https://launchpadlibrarian.net/777173611/juju-debug-log.txt
log of juju agent: https://launchpadlibrarian.net/777173886/machine-0-lxd-104.log
nova-api-wsgi.log: https://launchpadlibrarian.net/777173920/nova-api-wsgi.log
nova-conductor.log: https://launchpadlibrarian.net/777173960/nova-conductor.log
nova-scheduler.log: https://launchpadlibrarian.net/777173993/nova-scheduler.log
nova-manager.log: https://launchpadlibrarian.net/777174167/nova-manage.log
nova.conf: https://launchpadlibrarian.net/777176686/nova.conf
juju show unit nova-cloud-controller: https://launchpadlibrarian.net/777174454/juju-show-unit-nova-cloud-controller.txt