ERROR neutron OpenSSL.SSL.Error: certificate verify failed

charm OpenStack
, ,
1

I recently reissue the vault certificates after they expired and my cloud stooped working.

But now I am facing this problem that i don’t know how to solve. It seams nova has a problem


mchc@ubuntu:/data/git/openstack-cfg/client$ openstack server list --all-projects
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'neutronclient.common.exceptions.ServiceUnavailable'> (HTTP 500) (Request-ID: req-6f1ca795-bf57-448b-8c5a-be404415a2c0)

The neutron log error is :

geoint@MAAS-01:~$ juju ssh neutron-api/0
Last login: Tue Mar 26 15:30:49 2024 from 10.2.101.2
ubuntu@juju-5025f7-3-lxd-3:~$ tail /var/log/neutron/neutron-server.log
2024-03-26 15:37:04.990 131414 ERROR neutron   File "/usr/lib/python3/dist-packages/ovs/stream.py", line 811, in connect
2024-03-26 15:37:04.990 131414 ERROR neutron     self.socket.do_handshake()
2024-03-26 15:37:04.990 131414 ERROR neutron   File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1828, in do_handshake
2024-03-26 15:37:04.990 131414 ERROR neutron     self._raise_ssl_error(self._ssl, result)
2024-03-26 15:37:04.990 131414 ERROR neutron   File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1566, in _raise_ssl_error
2024-03-26 15:37:04.990 131414 ERROR neutron     _raise_current_error()
2024-03-26 15:37:04.990 131414 ERROR neutron   File "/usr/lib/python3/dist-packages/OpenSSL/_util.py", line 57, in exception_from_error_queue
2024-03-26 15:37:04.990 131414 ERROR neutron     raise exception_type(errors)
2024-03-26 15:37:04.990 131414 ERROR neutron OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
2024-03-26 15:37:04.990 131414 ERROR neutron 
2024-03-26 15:31:39.896 126255 ERROR neutron OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

al compute services are up:


mchc@ubuntu:/data/git/openstack-cfg/client$ openstack compute service list
+--------------------------------------+----------------+---------------------+----------+---------+-------+----------------------------+
| ID                                   | Binary         | Host                | Zone     | Status  | State | Updated At                 |
+--------------------------------------+----------------+---------------------+----------+---------+-------+----------------------------+
| 4224d70a-f8af-4b09-b5d5-ccb45287761d | nova-scheduler | juju-5025f7-1-lxd-4 | internal | enabled | up    | 2024-03-26T15:35:38.000000 |
| 5fdbb26b-7a2a-49e3-9cb1-e0ab9d50744d | nova-scheduler | juju-5025f7-2-lxd-4 | internal | enabled | up    | 2024-03-26T15:35:37.000000 |
| 5f465bb4-edfc-4499-b69c-4d208d5e2aae | nova-conductor | juju-5025f7-2-lxd-4 | internal | enabled | up    | 2024-03-26T15:35:42.000000 |
| 402b43ae-3ead-40b4-87af-c2bdc7392808 | nova-conductor | juju-5025f7-0-lxd-4 | internal | enabled | up    | 2024-03-26T15:35:42.000000 |
| d351d86f-87e7-4a7b-a8d3-df1c616229bf | nova-scheduler | juju-5025f7-0-lxd-4 | internal | enabled | up    | 2024-03-26T15:35:38.000000 |
| eaceda27-a46b-4b2e-911c-cf6c99dda28b | nova-conductor | juju-5025f7-1-lxd-4 | internal | enabled | up    | 2024-03-26T15:35:34.000000 |
| 8ec65e3e-9f74-4936-8612-335a446b7292 | nova-compute   | key-ox.maas         | MID-PCTY | enabled | up    | 2024-03-26T15:35:37.000000 |
| 23241ad8-a77a-408e-b291-baf957cc3bea | nova-compute   | stable-liger.maas   | MID-PCTY | enabled | up    | 2024-03-26T15:35:37.000000 |
| ec2cebd4-65e3-4c4d-84d8-135022727c8d | nova-compute   | star-koala.maas     | MID-PCTY | enabled | up    | 2024-03-26T15:35:37.000000 |
| bc29c17b-ad85-4d09-b7a4-1493b0601c24 | nova-compute   | liked-hermit.maas   | MID-PCTY | enabled | up    | 2024-03-26T15:35:37.000000 |
| ce6c5a79-3174-4b47-b365-a63a7b473428 | nova-compute   | exotic-goblin.maas  | MID-PCTY | enabled | up    | 2024-03-26T15:35:37.000000 |
| 3d42d217-daa3-4582-aabc-d5b3f3b47c98 | nova-compute   | clean-hog.maas      | MID-PCTY | enabled | up    | 2024-03-26T15:35:37.000000 |
| 4e022854-1511-4928-bdd5-920826069c12 | nova-compute   | calm-stag.maas      | MID-PCTY | enabled | up    | 2024-03-26T15:35:37.000000 |
| 967ea90e-417c-4d1d-9e5a-f25128a45ef6 | nova-compute   | pure-rabbit.maas    | MID-PCTY | enabled | up    | 2024-03-26T15:35:37.000000 |
| 385901eb-5237-448a-9db9-17a89498e2cf | nova-compute   | immune-mite.maas    | MID-PCTY | enabled | up    | 2024-03-26T15:35:37.000000 |
| 4d63ec87-378f-4e85-8ca0-d1e2960d91db | nova-compute   | vital-iguana.maas   | MID-PCTY | enabled | up    | 2024-03-26T15:35:37.000000 |
| 2b68b268-433e-4174-b175-2a3662de8808 | nova-compute   | loyal-glider.maas   | MID-PCTY | enabled | up    | 2024-03-26T15:35:37.000000 |
| 033ab76a-7436-422b-b0ea-9f044c563318 | nova-compute   | sure-pony.maas      | MID-PCTY | enabled | up    | 2024-03-26T15:35:37.000000 |
+--------------------------------------+----------------+---------------------+----------+---------+-------+----------------------------+
2

Could this be related?

geoint@MAAS-01:~$ juju run-action --wait neutron-api/0 security-checklist
unit-neutron-api-0:
  UnitId: neutron-api/0
  id: "1558"
  message: exit status 1
  results:
    ReturnCode: 1
    Stdout: |
      validate_enables_tls: FAIL (SSL should be enabled on neutron-api)
      validate_file_ownership: PASS
      validate_file_permissions: PASS
      validate_uses_keystone: PASS
      Skipping validate-uses-tls-for-glance because it isexcluded in audit config
      validate_uses_tls_for_keystone: PASS
    validate-enables-tls: FAIL - SSL should be enabled on neutron-api
    validate-file-ownership: PASS
    validate-file-permissions: PASS
    validate-uses-keystone: PASS
    validate-uses-tls-for-keystone: PASS
  status: failed
  timing:
    completed: 2024-03-26 17:08:30 +0000 UTC
    enqueued: 2024-03-26 17:08:29 +0000 UTC
    started: 2024-03-26 17:08:30 +0000 UTC
3

Apache log is:

ubuntu@juju-5025f7-3-lxd-3:~$ tail  /var/log/apache2/error.log
[Tue Mar 26 18:28:17.444913 2024] [proxy_http:error] [pid 14956:tid 140176508573440] [client 10.2.101.179:55002] AH01114: HTTP: failed to make connection to backend: localhost
[Tue Mar 26 18:28:35.999669 2024] [proxy:error] [pid 14956:tid 140176491788032] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:9676 (localhost) failed
[Tue Mar 26 18:28:35.999711 2024] [proxy_http:error] [pid 14956:tid 140176491788032] [client 10.2.101.179:37222] AH01114: HTTP: failed to make connection to backend: localhost
[Tue Mar 26 18:28:47.134862 2024] [proxy:error] [pid 14956:tid 140175896200960] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:9676 (localhost) failed
[Tue Mar 26 18:28:47.134906 2024] [proxy_http:error] [pid 14956:tid 140175896200960] [client 10.2.101.179:45892] AH01114: HTTP: failed to make connection to backend: localhost
[Tue Mar 26 18:28:56.044808 2024] [proxy:error] [pid 14957:tid 140175820699392] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:9676 (localhost) failed
[Tue Mar 26 18:28:56.044854 2024] [proxy_http:error] [pid 14957:tid 140175820699392] [client 10.2.101.179:55022] AH01114: HTTP: failed to make connection to backend: localhost
[Tue Mar 26 18:29:17.442598 2024] [proxy:error] [pid 14956:tid 140176173029120] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:9676 (localhost) failed
[Tue Mar 26 18:29:17.442643 2024] [proxy_http:error] [pid 14956:tid 140176173029120] [client 10.2.101.179:41844] AH01114: HTTP: failed to make connection to backend: localhost
[Tue Mar 26 18:29:36.358410 2024] [mpm_event:notice] [pid 309:tid 140176629136448] AH00491: caught SIGTERM, shutting down
4

I am not sure what created the problem but I try this and it solved it

juju remove-relation neutron-api:certificates vault:certificates
juju add-relation neutron-api:certificates vault:certificates


juju remove-relation neutron-api-plugin-ovn:certificates vault:certificates
juju add-relation neutron-api-plugin-ovn:certificates vault:certificates


juju remove-relation ovn-central:certificates vault:certificates
juju add-relation ovn-central:certificates vault:certificates

juju remove-relation ovn-chassis:certificates vault:certificates
juju add-relation ovn-chassis:certificates vault:certificates