Instance stuck at schedule state

, ,
1

Randomly some instances are in schedule state due to RabbitMQ issue. Check the below error.

How to restart the rabbitmq service using juju command?

grep -i "Error" /var/log/nova/nova-api-wsgi.log

2023-06-23 14:38:06.946 2850975 ERROR oslo.messaging._drivers.impl_rabbit [-] [c74a0690-5a96-40be-9bc7-b59c30370410] AMQP server on 10.85.5.102:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: ConnectionResetError: [Errno 104] Connection reset by peer
2023-06-23 14:40:42.542 2850973 ERROR oslo.messaging._drivers.impl_rabbit [-] [9263b7f2-731a-4d4a-a62e-789ae611334c] AMQP server on 10.85.5.102:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection
2023-06-23 14:41:02.647 2850973 ERROR oslo.messaging._drivers.impl_rabbit [-] [9263b7f2-731a-4d4a-a62e-789ae611334c] AMQP server on 10.85.5.102:5672 is unreachable: timed out. Trying again in 1 seconds.: socket.timeout: timed out
2023-06-23 14:43:18.255 2850974 ERROR oslo.messaging._drivers.impl_rabbit [-] [8e9335b9-e4d3-457e-a6f2-0eb569d409b4] AMQP server on 10.85.5.95:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: ConnectionResetError: [Errno 104] Connection reset by peer
2023-06-23 14:44:37.169 2850974 ERROR oslo.messaging._drivers.impl_rabbit [-] [8e9335b9-e4d3-457e-a6f2-0eb569d409b4] AMQP server on 10.85.5.95:5672 is unreachable: timed out. Trying again in 1 seconds.: socket.timeout: timed out
2023-06-23 14:51:18.518 2850974 ERROR oslo.messaging._drivers.impl_rabbit [-] [8e9335b9-e4d3-457e-a6f2-0eb569d409b4] AMQP server on 10.85.5.75:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: ConnectionResetError: [Errno 104] Connection reset by peer
2023-06-23 14:52:03.504 2850974 ERROR oslo.messaging._drivers.impl_rabbit [-] [8e9335b9-e4d3-457e-a6f2-0eb569d409b4] AMQP server on 10.85.5.75:5672 is unreachable: timed out. Trying again in 1 seconds.: socket.timeout: timed out
2023-06-23 14:53:11.145 2850975 ERROR oslo.messaging._drivers.impl_rabbit [-] [c74a0690-5a96-40be-9bc7-b59c30370410] AMQP server on 10.85.5.102:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection
2023-06-23 14:54:50.000 2850973 ERROR oslo.messaging._drivers.impl_rabbit [-] [9263b7f2-731a-4d4a-a62e-789ae611334c] AMQP server on 10.85.5.95:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection
2023-06-23 14:55:23.291 2850972 ERROR oslo.messaging._drivers.impl_rabbit [-] [cb3e3683-e404-4544-893f-c49716a79837] AMQP server on 10.85.5.75:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: ConnectionResetError: [Errno 104] Connection reset by peer
2023-06-23 14:57:02.122 2850974 ERROR oslo.messaging._drivers.impl_rabbit [-] [8e9335b9-e4d3-457e-a6f2-0eb569d409b4] AMQP server on 10.85.5.102:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection
2023-06-23 15:00:54.695 2850975 ERROR oslo.messaging._drivers.impl_rabbit [-] [c74a0690-5a96-40be-9bc7-b59c30370410] AMQP server on 10.85.5.102:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection
2023-06-23 15:00:54.777 2850974 ERROR oslo.messaging._drivers.impl_rabbit [-] [8e9335b9-e4d3-457e-a6f2-0eb569d409b4] AMQP server on 10.85.5.102:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection
2023-06-23 15:02:00.321 2850973 ERROR oslo.messaging._drivers.impl_rabbit [-] [9263b7f2-731a-4d4a-a62e-789ae611334c] AMQP server on 10.85.5.95:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: ConnectionResetError: [Errno 104] Connection reset by peer
2023-06-23 15:02:00.411 2850972 ERROR oslo.messaging._drivers.impl_rabbit [-] [cb3e3683-e404-4544-893f-c49716a79837] AMQP server on 10.85.5.75:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection
2023-06-23 15:04:45.931 2850974 ERROR oslo.messaging._drivers.impl_rabbit [-] [8e9335b9-e4d3-457e-a6f2-0eb569d409b4] AMQP server on 10.85.5.102:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: ConnectionResetError: [Errno 104] Connection reset by peer
2023-06-23 15:05:52.708 2850972 ERROR oslo.messaging._drivers.impl_rabbit [-] [cb3e3683-e404-4544-893f-c49716a79837] AMQP server on 10.85.5.75:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection
2023-06-23 15:05:52.736 2850975 ERROR oslo.messaging._drivers.impl_rabbit [-] [c74a0690-5a96-40be-9bc7-b59c30370410] AMQP server on 10.85.5.102:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection
``
2

@rrajasekar - hmm, do you know what the rabbit challenges are? I want to make sure we get to the root of the problem. It looks like the remote end is rejecting the connections from this snippet. What do the logs look like on the rabbit side?

There are some guides dealing with rabbitmq that may help in the OpenStack Charm Guide. I’m not entirely sure that you need these procedures, it kind of depends on what the rabbit state is. However, I mention this link to see if it helps you resolve/recover your situation.

3

@billy-olsen

RabbitMQ cluster status is normal, no partition found.

RabbitMQ closing connection issue may be due to rabbitmq file descriptor limit?

RabbitMQ server juju status

root@opc1-maas-openstack:~# juju status rabbitmq-server
Model      Controller       Cloud/Region      Version  SLA          Timestamp
openstack  maas-controller  maas-one/default  2.9.38   unsupported  05:39:28+05:30

App              Version  Status  Scale  Charm            Channel     Rev  Exposed  Message
rabbitmq-server  3.8.2    active      3  rabbitmq-server  3.9/stable  154  no       Unit is ready and clustered

Unit                Workload  Agent  Machine   Public address  Ports               Message
rabbitmq-server/0*  active    idle   3/lxd/10  10.85.5.75      5672/tcp,15672/tcp  Unit is ready and clustered
rabbitmq-server/1   active    idle   4/lxd/10  10.85.5.95      5672/tcp,15672/tcp  Unit is ready and clustered
rabbitmq-server/2   active    idle   5/lxd/10  10.85.5.102     5672/tcp,15672/tcp  Unit is ready and clustered

Machine   State    Address      Inst id                        Series  AZ       Message
3         started  10.85.2.41   controller-1-nx-opc1-b4-27-11  focal   default  Deployed
3/lxd/10  started  10.85.5.75   juju-717d4e-3-lxd-10           focal   default  Container started
4         started  10.85.2.69   controller-2-nx-opc1-b3-39-12  focal   default  Deployed
4/lxd/10  started  10.85.5.95   juju-717d4e-4-lxd-10           focal   default  Container started
5         started  10.85.2.62   controller-3-nx-opc1-b2-24-13  focal   default  Deployed
5/lxd/10  started  10.85.5.102  juju-717d4e-5-lxd-10           focal   default  Container started

RabbitMQ cluster status

juju ssh rabbitmq-server/0 sudo rabbitmqctl cluster_status

root@opc1-maas-openstack:~# juju ssh rabbitmq-server/0 sudo rabbitmqctl cluster_status
Cluster status of node rabbit@juju-717d4e-3-lxd-10 ...
Basics

Cluster name: rabbit@juju-717d4e-3-lxd-10.maas

Disk Nodes

rabbit@juju-717d4e-3-lxd-10
rabbit@juju-717d4e-4-lxd-10
rabbit@juju-717d4e-5-lxd-10

Running Nodes

rabbit@juju-717d4e-3-lxd-10
rabbit@juju-717d4e-4-lxd-10
rabbit@juju-717d4e-5-lxd-10

Versions

rabbit@juju-717d4e-3-lxd-10: RabbitMQ 3.8.2 on Erlang 22.2.7
rabbit@juju-717d4e-4-lxd-10: RabbitMQ 3.8.2 on Erlang 22.2.7
rabbit@juju-717d4e-5-lxd-10: RabbitMQ 3.8.2 on Erlang 22.2.7

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@juju-717d4e-3-lxd-10, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@juju-717d4e-3-lxd-10, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@juju-717d4e-3-lxd-10, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@juju-717d4e-4-lxd-10, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@juju-717d4e-4-lxd-10, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@juju-717d4e-4-lxd-10, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@juju-717d4e-5-lxd-10, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@juju-717d4e-5-lxd-10, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@juju-717d4e-5-lxd-10, interface: [::], port: 15672, protocol: http, purpose: HTTP API

Feature flags

Flag: drop_unroutable_metric, state: disabled
Flag: empty_basic_get_metric, state: disabled
Flag: implicit_default_bindings, state: enabled
Flag: quorum_queue, state: enabled
Flag: virtual_host_metadata, state: enabled
Connection to 10.85.5.75 closed.
root@opc1-maas-openstack:~#


juju ssh rabbitmq-server/1 sudo rabbitmqctl cluster_status

root@opc1-maas-openstack:~# juju ssh rabbitmq-server/1 sudo rabbitmqctl cluster_status
Cluster status of node rabbit@juju-717d4e-4-lxd-10 ...
Basics

Cluster name: rabbit@juju-717d4e-3-lxd-10.maas

Disk Nodes

rabbit@juju-717d4e-3-lxd-10
rabbit@juju-717d4e-4-lxd-10
rabbit@juju-717d4e-5-lxd-10

Running Nodes

rabbit@juju-717d4e-3-lxd-10
rabbit@juju-717d4e-4-lxd-10
rabbit@juju-717d4e-5-lxd-10

Versions

rabbit@juju-717d4e-3-lxd-10: RabbitMQ 3.8.2 on Erlang 22.2.7
rabbit@juju-717d4e-4-lxd-10: RabbitMQ 3.8.2 on Erlang 22.2.7
rabbit@juju-717d4e-5-lxd-10: RabbitMQ 3.8.2 on Erlang 22.2.7

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@juju-717d4e-3-lxd-10, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@juju-717d4e-3-lxd-10, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@juju-717d4e-3-lxd-10, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@juju-717d4e-4-lxd-10, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@juju-717d4e-4-lxd-10, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@juju-717d4e-4-lxd-10, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@juju-717d4e-5-lxd-10, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@juju-717d4e-5-lxd-10, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@juju-717d4e-5-lxd-10, interface: [::], port: 15672, protocol: http, purpose: HTTP API

Feature flags

Flag: drop_unroutable_metric, state: disabled
Flag: empty_basic_get_metric, state: disabled
Flag: implicit_default_bindings, state: enabled
Flag: quorum_queue, state: enabled
Flag: virtual_host_metadata, state: enabled
Connection to 10.85.5.95 closed.
root@opc1-maas-openstack:~#


juju ssh rabbitmq-server/2 sudo rabbitmqctl cluster_status

root@opc1-maas-openstack:~# juju ssh rabbitmq-server/2 sudo rabbitmqctl cluster_status
Cluster status of node rabbit@juju-717d4e-5-lxd-10 ...
Basics

Cluster name: rabbit@juju-717d4e-3-lxd-10.maas

Disk Nodes

rabbit@juju-717d4e-3-lxd-10
rabbit@juju-717d4e-4-lxd-10
rabbit@juju-717d4e-5-lxd-10

Running Nodes

rabbit@juju-717d4e-3-lxd-10
rabbit@juju-717d4e-4-lxd-10
rabbit@juju-717d4e-5-lxd-10

Versions

rabbit@juju-717d4e-3-lxd-10: RabbitMQ 3.8.2 on Erlang 22.2.7
rabbit@juju-717d4e-4-lxd-10: RabbitMQ 3.8.2 on Erlang 22.2.7
rabbit@juju-717d4e-5-lxd-10: RabbitMQ 3.8.2 on Erlang 22.2.7

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@juju-717d4e-3-lxd-10, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@juju-717d4e-3-lxd-10, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@juju-717d4e-3-lxd-10, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@juju-717d4e-4-lxd-10, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@juju-717d4e-4-lxd-10, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@juju-717d4e-4-lxd-10, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@juju-717d4e-5-lxd-10, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@juju-717d4e-5-lxd-10, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@juju-717d4e-5-lxd-10, interface: [::], port: 15672, protocol: http, purpose: HTTP API

Feature flags

Flag: drop_unroutable_metric, state: disabled
Flag: empty_basic_get_metric, state: disabled
Flag: implicit_default_bindings, state: enabled
Flag: quorum_queue, state: enabled
Flag: virtual_host_metadata, state: enabled
Connection to 10.85.5.102 closed.
root@opc1-maas-openstack:~#

Error from RabbitMQ log

2023-06-26 21:55:03.300 [error] <0.29169.404> closing AMQP connection <0.29169.404> (10.85.5.105:56306 -> 10.85.5.75:5672 - mod_wsgi:342588:8658715a-6e2d-46e0-9d61-cfa098636ffd):
2023-06-26 22:03:03.151 [error] <0.1718.405> closing AMQP connection <0.1718.405> (10.85.5.74:46566 -> 10.85.5.75:5672 - mod_wsgi:477349:63d58461-2572-4161-9cd5-28f2d502d337):
2023-06-26 22:03:25.369 [error] <0.1765.405> closing AMQP connection <0.1765.405> (10.85.5.74:56006 -> 10.85.5.75:5672 - mod_wsgi:477348:06f88160-7915-458c-9e96-3cf39608f776):
2023-06-26 22:06:04.960 [error] <0.2372.405> closing AMQP connection <0.2372.405> (10.85.5.105:54118 -> 10.85.5.75:5672 - mod_wsgi:342588:8658715a-6e2d-46e0-9d61-cfa098636ffd):
2023-06-26 22:13:20.967 [error] <0.3666.405> closing AMQP connection <0.3666.405> (10.85.5.74:59306 -> 10.85.5.75:5672 - mod_wsgi:477349:63d58461-2572-4161-9cd5-28f2d502d337):
2023-06-26 22:13:24.879 [error] <0.3682.405> closing AMQP connection <0.3682.405> (10.85.5.74:59320 -> 10.85.5.75:5672 - mod_wsgi:477348:06f88160-7915-458c-9e96-3cf39608f776):
2023-06-26 22:17:05.984 [error] <0.4485.405> closing AMQP connection <0.4485.405> (10.85.5.105:33024 -> 10.85.5.75:5672 - mod_wsgi:342588:8658715a-6e2d-46e0-9d61-cfa098636ffd):
2023-06-26 22:23:22.345 [error] <0.5637.405> closing AMQP connection <0.5637.405> (10.85.5.74:38340 -> 10.85.5.75:5672 - mod_wsgi:477349:63d58461-2572-4161-9cd5-28f2d502d337):
2023-06-26 22:28:02.984 [error] <0.6509.405> closing AMQP connection <0.6509.405> (10.85.5.74:38420 -> 10.85.5.75:5672 - mod_wsgi:477348:06f88160-7915-458c-9e96-3cf39608f776):
2023-06-26 22:33:03.720 [error] <0.7466.405> closing AMQP connection <0.7466.405> (10.85.5.105:41002 -> 10.85.5.75:5672 - mod_wsgi:342588:8658715a-6e2d-46e0-9d61-cfa098636ffd):
2023-06-26 22:33:20.067 [error] <0.7493.405> closing AMQP connection <0.7493.405> (10.85.5.74:45964 -> 10.85.5.75:5672 - mod_wsgi:477349:63d58461-2572-4161-9cd5-28f2d502d337):
2023-06-26 22:33:23.738 [error] <0.7507.405> closing AMQP connection <0.7507.405> (10.85.5.74:45974 -> 10.85.5.75:5672 - mod_wsgi:477348:06f88160-7915-458c-9e96-3cf39608f776):
2023-06-26 22:43:21.381 [error] <0.9398.405> closing AMQP connection <0.9398.405> (10.85.5.105:39706 -> 10.85.5.75:5672 - mod_wsgi:342588:8658715a-6e2d-46e0-9d61-cfa098636ffd):
2023-06-26 22:44:03.006 [error] <0.9514.405> closing AMQP connection <0.9514.405> (10.85.5.74:55010 -> 10.85.5.75:5672 - mod_wsgi:477349:63d58461-2572-4161-9cd5-28f2d502d337):
2023-06-26 22:44:06.455 [error] <0.9529.405> closing AMQP connection <0.9529.405> (10.85.5.74:55024 -> 10.85.5.75:5672 - mod_wsgi:477348:06f88160-7915-458c-9e96-3cf39608f776):
2023-06-26 22:53:23.666 [error] <0.10888.405> closing AMQP connection <0.10888.405> (10.85.5.105:59834 -> 10.85.5.75:5672 - mod_wsgi:342588:8658715a-6e2d-46e0-9d61-cfa098636ffd):
2023-06-26 22:55:04.957 [error] <0.11218.405> closing AMQP connection <0.11218.405> (10.85.5.74:42400 -> 10.85.5.75:5672 - mod_wsgi:477349:63d58461-2572-4161-9cd5-28f2d502d337):
2023-06-26 23:03:03.448 [error] <0.12924.405> closing AMQP connection <0.12924.405> (10.85.5.74:43184 -> 10.85.5.75:5672 - mod_wsgi:477348:06f88160-7915-458c-9e96-3cf39608f776):
2023-06-26 23:03:27.983 [error] <0.21024.398> closing AMQP connection <0.21024.398> (10.85.5.105:44980 -> 10.85.5.75:5672 - mod_wsgi:342588:8658715a-6e2d-46e0-9d61-cfa098636ffd):
2023-06-26 23:06:06.214 [error] <0.13529.405> closing AMQP connection <0.13529.405> (10.85.5.74:53598 -> 10.85.5.75:5672 - mod_wsgi:477349:63d58461-2572-4161-9cd5-28f2d502d337):
2023-06-26 23:13:21.231 [error] <0.14840.405> closing AMQP connection <0.14840.405> (10.85.5.74:56722 -> 10.85.5.75:5672 - mod_wsgi:477348:06f88160-7915-458c-9e96-3cf39608f776):
2023-06-26 23:17:03.256 [error] <0.15660.405> closing AMQP connection <0.15660.405> (10.85.5.105:47306 -> 10.85.5.75:5672 - mod_wsgi:342588:8658715a-6e2d-46e0-9d61-cfa098636ffd):
2023-06-26 23:23:03.224 [error] <0.16831.405> closing AMQP connection <0.16831.405> (10.85.5.74:44066 -> 10.85.5.75:5672 - mod_wsgi:477349:63d58461-2572-4161-9cd5-28f2d502d337):
2023-06-26 23:23:22.928 [error] <0.16862.405> closing AMQP connection <0.16862.405> (10.85.5.74:41106 -> 10.85.5.75:5672 - mod_wsgi:477348:06f88160-7915-458c-9e96-3cf39608f776):
2023-06-26 23:28:04.312 [error] <0.17618.405> closing AMQP connection <0.17618.405> (10.85.5.105:39092 -> 10.85.5.75:5672 - mod_wsgi:342588:8658715a-6e2d-46e0-9d61-cfa098636ffd):
2023-06-26 23:33:14.821 [error] <0.18631.405> closing AMQP connection <0.18631.405> (10.85.5.74:54700 -> 10.85.5.75:5672 - mod_wsgi:477349:63d58461-2572-4161-9cd5-28f2d502d337):
2023-06-26 23:33:18.743 [error] <0.18646.405> closing AMQP connection <0.18646.405> (10.85.5.74:35996 -> 10.85.5.75:5672 - mod_wsgi:477348:06f88160-7915-458c-9e96-3cf39608f776):
2023-06-26 23:43:03.547 [error] <0.20565.405> closing AMQP connection <0.20565.405> (10.85.5.105:46312 -> 10.85.5.75:5672 - mod_wsgi:342588:8658715a-6e2d-46e0-9d61-cfa098636ffd):
2023-06-26 23:43:20.993 [error] <0.20582.405> closing AMQP connection <0.20582.405> (10.85.5.74:42318 -> 10.85.5.75:5672 - mod_wsgi:477349:63d58461-2572-4161-9cd5-28f2d502d337):
2023-06-26 23:43:24.958 [error] <0.20595.405> closing AMQP connection <0.20595.405> (10.85.5.74:42326 -> 10.85.5.75:5672 - mod_wsgi:477348:06f88160-7915-458c-9e96-3cf39608f776):
2023-06-26 23:53:21.843 [error] <0.22570.405> closing AMQP connection <0.22570.405> (10.85.5.105:43276 -> 10.85.5.75:5672 - mod_wsgi:342588:8658715a-6e2d-46e0-9d61-cfa098636ffd):
2023-06-26 23:53:23.624 [error] <0.22569.405> closing AMQP connection <0.22569.405> (10.85.5.74:37996 -> 10.85.5.75:5672 - mod_wsgi:477349:63d58461-2572-4161-9cd5-28f2d502d337):
2023-06-26 23:55:02.654 [error] <0.22885.405> closing AMQP connection <0.22885.405> (10.85.5.74:37262 -> 10.85.5.75:5672 - mod_wsgi:477348:06f88160-7915-458c-9e96-3cf39608f776):
2023-06-27 13:39:54.080 [error] <0.21439.328> closing AMQP connection <0.21439.328> (10.85.5.59:34474 -> 10.85.5.75:5672 - nova-compute:171622:476e08c8-5618-4048-bcb6-679c960f55f3):
2023-06-27 18:24:24.203 [error] <0.13485.0> closing AMQP connection <0.13485.0> (10.85.5.52:58082 -> 10.85.5.75:5672 - nova-api-metadata:199250:6559a46b-cc7b-4ac0-88b4-73845b628a01):
2023-06-27 23:56:27.132 [error] <0.5625.414> closing AMQP connection <0.5625.414> (10.85.5.74:59484 -> 10.85.5.75:5672 - mod_wsgi:1084178:f4ca6cfd-d097-4060-bdcb-8934775bbb42):