OpenStack SSLEOFError: EOF occurred in violation of protocol

, , , ,
1

Hi Team,

I am facing the below SSL error. Any help on this?

HA setup with 3 controller node + 3 CEPH node + 20 compute noes
OpenStack version: Yoga
Host OS:  Ubuntu 20.04
Juju version: 2.9.37

2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [req-ab81de29-ccc4-4af5-851b-ae1ec48db6b0 0e42367c47734932b9a535a27314547e 75a4f053a6b6415898119a29debcfe76 - aca58c7f17af48ef8d7aef60764250dc aca58c7f17af48ef8d7aef60764250dc] [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6] Failed to build and run instance: keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://10.91.6.27:9696/v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55: HTTPSConnectionPool(host='10.91.6.27', port=9696): Max retries exceeded with url: /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2635)')))
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6] Traceback (most recent call last):
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     httplib_response = self._make_request(
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     six.raise_from(e, None)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "<string>", line 3, in raise_from
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 416, in _make_request
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     httplib_response = conn.getresponse()
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     response.begin()
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3.8/http/client.py", line 316, in begin
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     version, status, reason = self._read_status()
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3.8/http/client.py", line 277, in _read_status
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3.8/socket.py", line 669, in readinto
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     return self._sock.recv_into(b)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 252, in recv_into
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     return self._base_recv(nbytes, flags, into=True, buffer_=buffer)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 267, in _base_recv
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     read = self.read(nbytes, buffer_)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 186, in read
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     return self._call_trampolining(
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 161, in _call_trampolining
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     return func(*a, **kw)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3.8/ssl.py", line 1099, in read
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     return self._sslobj.read(len, buffer)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6] ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2635)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6] During handling of the above exception, another exception occurred:
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6] Traceback (most recent call last):
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     resp = conn.urlopen(
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     retries = retries.increment(
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     raise MaxRetryError(_pool, url, error or ResponseError(cause))
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6] urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.91.6.27', port=9696): Max retries exceeded with url: /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2635)')))
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6] During handling of the above exception, another exception occurred:
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6] Traceback (most recent call last):
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1022, in _send_request
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     resp = self.session.request(method, url, **kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     resp = self.send(prep, **send_kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     r = adapter.send(request, **kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 514, in send
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     raise SSLError(e, request=request)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6] requests.exceptions.SSLError: HTTPSConnectionPool(host='10.91.6.27', port=9696): Max retries exceeded with url: /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2635)')))
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6] During handling of the above exception, another exception occurred:
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6] Traceback (most recent call last):
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2503, in _build_and_run_instance
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     self.driver.spawn(context, instance, image_meta,
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4329, in spawn
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     xml = self._get_guest_xml(context, instance, network_info,
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7288, in _get_guest_xml
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     network_info_str = str(network_info)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/network/model.py", line 620, in __str__
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     return self._sync_wrapper(fn, *args, **kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/network/model.py", line 603, in _sync_wrapper
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     self.wait()
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/network/model.py", line 635, in wait
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     self[:] = self._gt.wait()
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/eventlet/greenthread.py", line 181, in wait
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     return self._exit_event.wait()
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/eventlet/event.py", line 125, in wait
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     result = hub.switch()
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/eventlet/hubs/hub.py", line 313, in switch
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     return self.greenlet.switch()
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/eventlet/greenthread.py", line 221, in main
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     result = function(*args, **kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/utils.py", line 656, in context_wrapper
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     return func(*args, **kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 1890, in _allocate_network_async
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     raise e
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 1868, in _allocate_network_async
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     nwinfo = self.network_api.allocate_for_instance(
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1215, in allocate_for_instance
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     created_port_ids = self._update_ports_for_instance(
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1357, in _update_ports_for_instance
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     vif.destroy()
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     self.force_reraise()
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     raise self.value
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1326, in _update_ports_for_instance
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     updated_port = self._update_port(
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 582, in _update_port
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     port_response = port_client.update_port(port_id, port_req_body)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 197, in wrapper
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     ret = obj(*args, **kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 826, in update_port
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     return self._update_resource(self.port_path % (port), body=body,
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 197, in wrapper
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     ret = obj(*args, **kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 2520, in _update_resource
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     return self.put(path, **kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 197, in wrapper
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     ret = obj(*args, **kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 365, in put
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     return self.retry_request("PUT", action, body=body,
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 197, in wrapper
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     ret = obj(*args, **kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 333, in retry_request
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     return self.do_request(method, action, body=body,
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 197, in wrapper
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     ret = obj(*args, **kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 284, in do_request
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     resp, replybody = self.httpclient.do_request(action, method, body=body,
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/neutronclient/client.py", line 342, in do_request
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     return self.request(url, method, **kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/neutronclient/client.py", line 330, in request
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     resp = super(SessionClient, self).request(*args, **kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     return self.session.request(url, method, **kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 931, in request
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     resp = send(**kwargs)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]   File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1026, in _send_request
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6]     raise exceptions.SSLError(msg)
2023-05-11 05:48:28.703 89372 ERROR nova.compute.manager [instance: 53e8f644-d9b1-472a-9a01-f41f9e2f89b6] keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://10.91.6.27:9696/v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55: HTTPSConnectionPool(host='10.91.6.27', port=9696): Max retries exceeded with url: /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2635)')))
2

Hi Rajasekar,

[...] requests.exceptions.SSLError: HTTPSConnectionPool(host='10.91.6.27', port=9696): Max retries exceeded with url: /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2635)')))

This looks like an issue on the neutron-server side, the connection seems to have been closed unexpectedly from that end, I suggest you to grep /var/log/neutron/neutron-server.log, look for errors and/or log lines that contain the port id (e8a5f31d-f560-476a-bbd3-1b8f6e428a55) or the instance id.

3

@freyes

grep output from neutron-server.log

I am facing this issue randomly.

from neutron-server/0 unit

root@juju-ca6801-3-lxd-5:~# grep -ir "e8a5f31d-f560-476a-bbd3-1b8f6e428a55" /var/log/neutron/neutron-server.log
2023-05-11 05:46:27.516 119588 INFO neutron.db.ovn_revision_numbers_db [req-3e03e13c-6fa2-4c49-9130-3fa2c4b2810c 0e42367c47734932b9a535a27314547e 75a4f053a6b6415898119a29debcfe76 - aca58c7f17af48ef8d7aef60764250dc aca58c7f17af48ef8d7aef60764250dc] Successfully bumped revision number for resource e8a5f31d-f560-476a-bbd3-1b8f6e428a55 (type: ports) to 1
2023-05-11 05:46:40.924 119590 INFO neutron.wsgi [req-d7d07337-4d80-433e-9b05-6ad23c406f7d 0e42367c47734932b9a535a27314547e 75a4f053a6b6415898119a29debcfe76 - aca58c7f17af48ef8d7aef60764250dc aca58c7f17af48ef8d7aef60764250dc] 10.91.8.53,127.0.0.1 "GET /v2.0/ports?id=e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 924 time: 0.0456002
2023-05-11 05:46:42.882 119587 INFO neutron.wsgi [req-1b4ad436-5a30-4759-a38e-c371e0207ee9 0e42367c47734932b9a535a27314547e 75a4f053a6b6415898119a29debcfe76 - aca58c7f17af48ef8d7aef60764250dc aca58c7f17af48ef8d7aef60764250dc] 10.91.6.59,127.0.0.1 "GET /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 921 time: 0.2974923
2023-05-11 05:49:46.196 119587 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-86a7e559-b47b-4167-8dc3-b03494ab5056 - - - - -] OVN reports status up for port: e8a5f31d-f560-476a-bbd3-1b8f6e428a55
2023-05-11 05:49:46.391 119587 INFO neutron.notifiers.nova [-] Nova event matching ['req-15522d15-6401-4959-9434-429530c48ca9'] response: {'server_uuid': '53e8f644-d9b1-472a-9a01-f41f9e2f89b6', 'name': 'network-vif-plugged', 'status': 'completed', 'tag': 'e8a5f31d-f560-476a-bbd3-1b8f6e428a55', 'code': 200}
2023-05-11 05:49:46.598 119587 INFO neutron.db.ovn_revision_numbers_db [req-86a7e559-b47b-4167-8dc3-b03494ab5056 - - - - -] Successfully bumped revision number for resource e8a5f31d-f560-476a-bbd3-1b8f6e428a55 (type: ports) to 8
2023-05-11 05:49:48.447 119587 INFO neutron.notifiers.nova [-] Nova event matching ['req-38597c65-6b63-4ed3-8ece-2268386d7594'] response: {'server_uuid': '53e8f644-d9b1-472a-9a01-f41f9e2f89b6', 'name': 'network-vif-plugged', 'status': 'completed', 'tag': 'e8a5f31d-f560-476a-bbd3-1b8f6e428a55', 'code': 200}
2023-05-11 05:58:44.085 119587 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-86a7e559-b47b-4167-8dc3-b03494ab5056 - - - - -] OVN reports status down for port: e8a5f31d-f560-476a-bbd3-1b8f6e428a55
2023-05-11 05:58:44.404 119587 INFO neutron.notifiers.nova [-] Nova event matching ['req-0e9e719f-7daa-4b31-990f-da6f706c3ff2'] response: {'server_uuid': '53e8f644-d9b1-472a-9a01-f41f9e2f89b6', 'name': 'network-vif-unplugged', 'status': 'completed', 'tag': 'e8a5f31d-f560-476a-bbd3-1b8f6e428a55', 'code': 200}
2023-05-11 05:58:44.657 119587 INFO neutron.db.ovn_revision_numbers_db [req-86a7e559-b47b-4167-8dc3-b03494ab5056 - - - - -] Successfully bumped revision number for resource e8a5f31d-f560-476a-bbd3-1b8f6e428a55 (type: ports) to 9
2023-05-11 05:58:46.560 119587 INFO neutron.notifiers.nova [-] Nova event matching ['req-c7eb0e5f-6ac9-4de0-b464-2f13eb5d337f'] response: {'server_uuid': '53e8f644-d9b1-472a-9a01-f41f9e2f89b6', 'name': 'network-vif-plugged', 'status': 'completed', 'tag': 'e8a5f31d-f560-476a-bbd3-1b8f6e428a55', 'code': 200}

from neutron-server/1 unit

ubuntu@juju-ca6801-4-lxd-5:~$ sudo su -
root@juju-ca6801-4-lxd-5:~# grep -ir "e8a5f31d-f560-476a-bbd3-1b8f6e428a55" /var/log/neutron/neutron-server.log
2023-05-11 05:46:41.655 454092 INFO neutron.wsgi [req-121f47dc-06b5-4bd1-af56-39bc7d73a429 0e42367c47734932b9a535a27314547e 75a4f053a6b6415898119a29debcfe76 - aca58c7f17af48ef8d7aef60764250dc aca58c7f17af48ef8d7aef60764250dc] 10.91.6.59,127.0.0.1 "GET /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 921 time: 0.2596533
2023-05-11 05:46:41.793 454091 INFO neutron.wsgi [req-a3d257e5-ddc2-41d3-8a3f-31e490b02bf3 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] 10.91.6.59,127.0.0.1 "GET /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55?fields=binding%3Avnic_type&fields=binding%3Aprofile&fields=network_id&fields=resource_request&fields=numa_affinity_policy&fields=device_profile HTTP/1.1" status: 200  len: 311 time: 0.0394304
2023-05-11 05:46:42.353 454091 INFO neutron.wsgi [req-d8d5bde7-0746-4dc5-b554-d2efedf4c068 0e42367c47734932b9a535a27314547e 75a4f053a6b6415898119a29debcfe76 - aca58c7f17af48ef8d7aef60764250dc aca58c7f17af48ef8d7aef60764250dc] 10.91.6.59,127.0.0.1 "GET /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 921 time: 0.2420909
2023-05-11 05:50:06.366 454090 INFO neutron.wsgi [req-c26ee0c4-41fd-4945-82da-7e0988746597 0e42367c47734932b9a535a27314547e 75a4f053a6b6415898119a29debcfe76 - aca58c7f17af48ef8d7aef60764250dc aca58c7f17af48ef8d7aef60764250dc] 10.91.8.53,127.0.0.1 "GET /v2.0/ports?id=e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 974 time: 0.0554476

from neutron-server/2 unit

root@juju-ca6801-5-lxd-5:~# grep -ir "e8a5f31d-f560-476a-bbd3-1b8f6e428a55" /var/log/neutron/neutron-server.log
2023-05-11 05:46:33.025 406216 INFO neutron.wsgi [req-82d55052-f126-4cfd-aaf9-cd184ef04d28 0e42367c47734932b9a535a27314547e 75a4f053a6b6415898119a29debcfe76 - aca58c7f17af48ef8d7aef60764250dc aca58c7f17af48ef8d7aef60764250dc] 10.91.8.53,127.0.0.1 "GET /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 921 time: 0.3243318
2023-05-11 05:46:33.213 406216 INFO neutron.wsgi [req-377c309d-57f2-40e8-98e6-b2173421f120 0e42367c47734932b9a535a27314547e 75a4f053a6b6415898119a29debcfe76 - aca58c7f17af48ef8d7aef60764250dc aca58c7f17af48ef8d7aef60764250dc] 10.91.8.53,127.0.0.1 "GET /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 921 time: 0.1838448
2023-05-11 05:46:45.123 406218 INFO neutron.wsgi [req-f2edf734-3648-43b1-abec-45ad8383a3f7 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] 10.91.6.59,127.0.0.1 "GET /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 1017 time: 0.0309141
2023-05-11 05:48:17.220 406217 INFO neutron.wsgi [req-09ca24c8-bb4c-45a2-8e93-d752fbe81b2b 0e42367c47734932b9a535a27314547e 75a4f053a6b6415898119a29debcfe76 - aca58c7f17af48ef8d7aef60764250dc aca58c7f17af48ef8d7aef60764250dc] 10.91.6.59,127.0.0.1 "GET /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55?fields=binding%3Aprofile&fields=network_id HTTP/1.1" status: 200  len: 260 time: 0.1272361
2023-05-11 05:48:27.966 406217 INFO neutron.db.ovn_revision_numbers_db [req-cb0df842-d9c8-40bc-aef7-23790272eba1 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] Successfully bumped revision number for resource e8a5f31d-f560-476a-bbd3-1b8f6e428a55 (type: ports) to 2
2023-05-11 05:48:27.980 406217 INFO neutron.wsgi [req-cb0df842-d9c8-40bc-aef7-23790272eba1 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] 10.91.6.59,127.0.0.1 "PUT /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 1017 time: 10.6474795
2023-05-11 05:48:30.256 406217 INFO neutron.wsgi [req-2cbdd299-344c-4530-93a4-9857da1912c8 0e42367c47734932b9a535a27314547e 75a4f053a6b6415898119a29debcfe76 - aca58c7f17af48ef8d7aef60764250dc aca58c7f17af48ef8d7aef60764250dc] 10.91.6.59,127.0.0.1 "GET /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55?fields=binding%3Aprofile&fields=network_id HTTP/1.1" status: 200  len: 260 time: 0.1438255
2023-05-11 05:49:19.126 406217 INFO neutron.db.ovn_revision_numbers_db [req-1b420360-3fc7-477d-8033-bad676e23108 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] Successfully bumped revision number for resource e8a5f31d-f560-476a-bbd3-1b8f6e428a55 (type: ports) to 3
2023-05-11 05:49:19.158 406217 INFO neutron.wsgi [req-1b420360-3fc7-477d-8033-bad676e23108 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] 10.91.6.59,127.0.0.1 "PUT /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 1017 time: 48.7633669
2023-05-11 05:49:21.073 406217 INFO neutron.wsgi [req-b88410d9-3f2c-4ede-83e0-e12d6d764c29 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] 10.91.6.59,127.0.0.1 "GET /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 1017 time: 0.0462987
2023-05-11 05:49:36.060 406218 INFO neutron.db.ovn_revision_numbers_db [req-7972ac10-0d51-405b-8f62-3454a56273ea 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] Successfully bumped revision number for resource e8a5f31d-f560-476a-bbd3-1b8f6e428a55 (type: ports) to 4
2023-05-11 05:49:36.819 406218 INFO neutron.wsgi [req-7972ac10-0d51-405b-8f62-3454a56273ea 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] 10.91.6.59,127.0.0.1 "PUT /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 1125 time: 171.1873279
2023-05-11 05:49:37.717 406217 INFO neutron.db.ovn_revision_numbers_db [req-5b146e65-db1c-4e36-bf3f-48c42ec88e60 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] Successfully bumped revision number for resource e8a5f31d-f560-476a-bbd3-1b8f6e428a55 (type: ports) to 6
2023-05-11 05:49:38.142 406217 INFO neutron.wsgi [req-5b146e65-db1c-4e36-bf3f-48c42ec88e60 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] 10.91.6.59,127.0.0.1 "PUT /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 1124 time: 16.7647007
2023-05-11 05:49:38.247 406217 INFO neutron.notifiers.nova [-] Nova event matching ['req-c2f279e4-f1b2-49a3-a1a7-82f4a05ef957'] response: {'name': 'network-changed', 'server_uuid': '53e8f644-d9b1-472a-9a01-f41f9e2f89b6', 'tag': 'e8a5f31d-f560-476a-bbd3-1b8f6e428a55', 'status': 'completed', 'code': 200}
2023-05-11 05:49:38.484 406217 INFO neutron.wsgi [req-12092d31-027f-43d0-89cb-d1a5dee50407 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] 10.91.6.59,127.0.0.1 "GET /v2.0/floatingips?fixed_ip_address=10.25.123.15&port_id=e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 217 time: 0.0918798
2023-05-11 05:49:38.694 406218 INFO neutron.notifiers.nova [-] Nova event matching ['req-290e2818-4f1b-45aa-a609-cc6b03569397'] response: {'name': 'network-changed', 'server_uuid': '53e8f644-d9b1-472a-9a01-f41f9e2f89b6', 'tag': 'e8a5f31d-f560-476a-bbd3-1b8f6e428a55', 'status': 'completed', 'code': 200}
2023-05-11 05:49:39.394 406217 INFO neutron.wsgi [req-c76178b8-00f3-4d6f-bed7-13218fd17d89 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] 10.91.6.59,127.0.0.1 "GET /v2.0/floatingips?fixed_ip_address=10.25.123.15&port_id=e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 217 time: 0.0362051
2023-05-11 05:49:40.177 406217 INFO neutron.wsgi [req-de354bec-4362-4b54-93b5-1bbdf868b690 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] 10.91.6.59,127.0.0.1 "GET /v2.0/floatingips?fixed_ip_address=10.25.123.15&port_id=e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 217 time: 0.0315993
2023-05-11 05:58:36.331 406219 INFO neutron.wsgi [req-25090a59-3ee6-467d-9beb-a996ab7040be 0e42367c47734932b9a535a27314547e 75a4f053a6b6415898119a29debcfe76 - aca58c7f17af48ef8d7aef60764250dc aca58c7f17af48ef8d7aef60764250dc] 10.91.8.53,127.0.0.1 "GET /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 971 time: 0.4762948
2023-05-11 05:58:46.642 406218 INFO neutron.wsgi [req-ca54aa47-5a0a-44d1-8b9f-5c3438413c45 0e42367c47734932b9a535a27314547e 75a4f053a6b6415898119a29debcfe76 - aca58c7f17af48ef8d7aef60764250dc aca58c7f17af48ef8d7aef60764250dc] 10.91.6.59,127.0.0.1 "GET /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55?fields=binding%3Aprofile&fields=network_id HTTP/1.1" status: 200  len: 260 time: 0.3491280
2023-05-11 05:58:47.963 406218 INFO neutron.db.ovn_revision_numbers_db [req-6b1a0636-12a0-4333-aa3b-d9d488e1f390 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] Successfully bumped revision number for resource e8a5f31d-f560-476a-bbd3-1b8f6e428a55 (type: ports) to 10
2023-05-11 05:58:48.015 406218 INFO neutron.wsgi [req-6b1a0636-12a0-4333-aa3b-d9d488e1f390 0e1219c9abd848308a340b1c38cf66aa ab795740239f453bb22ae47aa5fe96ae - 07df3f7d62334d9f8b2eaab34a08cd85 07df3f7d62334d9f8b2eaab34a08cd85] 10.91.6.59,127.0.0.1 "PUT /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 200  len: 1018 time: 1.1857786
2023-05-11 05:59:28.869 406219 INFO neutron.wsgi [req-db097f82-3848-4825-b340-ce97b8920311 0e42367c47734932b9a535a27314547e 75a4f053a6b6415898119a29debcfe76 - aca58c7f17af48ef8d7aef60764250dc aca58c7f17af48ef8d7aef60764250dc] 10.91.8.53,127.0.0.1 "DELETE /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 204  len: 173 time: 2.7375395
2023-05-11 05:59:33.957 406219 INFO neutron.wsgi [req-461b6653-d8fd-483f-90fd-8995825c08ee 0e42367c47734932b9a535a27314547e 75a4f053a6b6415898119a29debcfe76 - aca58c7f17af48ef8d7aef60764250dc aca58c7f17af48ef8d7aef60764250dc] 10.91.8.53,127.0.0.1 "GET /v2.0/ports/e8a5f31d-f560-476a-bbd3-1b8f6e428a55 HTTP/1.1" status: 404  len: 338 time: 0.0485771
4

Found timeout and reconnect in neutron-server systemd status. Its a normal one?

root@juju-ca6801-3-lxd-5:~# systemctl status neutron-server.service
● neutron-server.service - OpenStack Neutron Server
     Loaded: loaded (/lib/systemd/system/neutron-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-05-11 02:40:32 UTC; 11h ago
       Docs: man:neutron-server(1)
   Main PID: 119491 (/usr/bin/python)
      Tasks: 27 (limit: 309252)
     Memory: 1.6G
     CGroup: /system.slice/neutron-server.service
             ├─119491 /usr/bin/python3 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log-file=/var/log/neutron/neutro>
             ├─119587 neutron-server: api worker (/usr/bin/python3 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log->
             ├─119588 neutron-server: api worker (/usr/bin/python3 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log->
             ├─119589 neutron-server: api worker (/usr/bin/python3 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log->
             ├─119590 neutron-server: api worker (/usr/bin/python3 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log->
             ├─123208 neutron-server: rpc worker (/usr/bin/python3 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log->
             ├─123209 neutron-server: rpc worker (/usr/bin/python3 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log->
             ├─123210 neutron-server: rpc worker (/usr/bin/python3 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log->
             ├─123211 neutron-server: rpc worker (/usr/bin/python3 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log->
             ├─123212 neutron-server: rpc worker (/usr/bin/python3 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log->
             ├─123213 neutron-server: MaintenanceWorker (/usr/bin/python3 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini>
             └─123214 neutron-server: rpc worker (/usr/bin/python3 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log->

May 11 14:15:23 juju-ca6801-3-lxd-5 ovsdb-client[789507]: ovs|00007|reconnect|INFO|ssl:10.91.6.73:16642: connecting...
May 11 14:15:23 juju-ca6801-3-lxd-5 ovsdb-client[789507]: ovs|00008|reconnect|INFO|ssl:10.91.6.73:16642: connected
May 11 14:15:39 juju-ca6801-3-lxd-5 ovsdb-client[789690]: ovs|00001|reconnect|INFO|ssl:10.91.6.49:16642: connecting...
May 11 14:15:39 juju-ca6801-3-lxd-5 ovsdb-client[789690]: ovs|00002|reconnect|INFO|ssl:10.91.6.49:16642: connected
May 11 14:15:39 juju-ca6801-3-lxd-5 ovsdb-client[789690]: ovs|00003|reconnect|INFO|ssl:10.91.6.49:16642: connection attempt timed out
May 11 14:15:39 juju-ca6801-3-lxd-5 ovsdb-client[789690]: ovs|00004|reconnect|INFO|ssl:10.91.6.80:16642: connecting...
May 11 14:15:39 juju-ca6801-3-lxd-5 ovsdb-client[789690]: ovs|00005|reconnect|INFO|ssl:10.91.6.80:16642: connected
May 11 14:15:39 juju-ca6801-3-lxd-5 ovsdb-client[789690]: ovs|00006|reconnect|INFO|ssl:10.91.6.80:16642: connection attempt timed out
May 11 14:15:39 juju-ca6801-3-lxd-5 ovsdb-client[789690]: ovs|00007|reconnect|INFO|ssl:10.91.6.73:16642: connecting...
May 11 14:15:39 juju-ca6801-3-lxd-5 ovsdb-client[789690]: ovs|00008|reconnect|INFO|ssl:10.91.6.73:16642: connected
5

Found timeout and reconnect in neutron-server systemd status. Its a normal one?

No, it’s not normal, when I’ve seen this behavior is commonly related to performance issues. This being a random issue, I would enable debug mode (juju config neutron-api debug=True) and wait for it to happen again to understand what neutron-server is doing before getting into this problem, there may be issues with the size of your cloud (e.g. not enough workers, the database not fast enough, etc), you would benefit from having nova-compute in debug mode as well, so you can correlate what nova-compute is requesting when neutron-server didn’t complete the request.