The placement service for keystone.openstack.maas:RegionOne exists but does not have any supported versions

, ,
1

I am currently unable to create or delete instances in my OpenStack Cloud (the cloud was working properly before, for a long time) . Everything seams to be working properly acording to juju status, but I am unable to create instances, they get stuck in Building status and when I try t delete an instance I got the following error.

mchc@ubuntu:/data/git/openstack-cfg/client$ openstack server delete --wait 45f3f3ff-8f72-4953-8894-3eab2c3affe8
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'openstack.exceptions.NotSupported'> (HTTP 500) (Request-ID: req-f3eec74d-a733-4104-8282-801deb88136d)

Login into nova-cloud-controler I can see the following error for nova-conductor.log

2023-03-22 17:59:05.006 1201203 ERROR nova openstack.exceptions.NotSupported: The placement service for keystone.openstack.maas:RegionOne exists but does not have any supported versions.

This error for nova-scheduler.log

2023-03-22 18:10:26.812 1208781 WARNING keystoneauth.discover [req-cef0bf8c-0590-445c-ae6f-b019724166cd - - - - -] Failed to contact the endpoint at https://placement.openstack.maas:8778 for discovery. Fallback to using that endpoint as the base url.
2023-03-22 18:10:26.828 1208781 WARNING keystoneauth.discover [req-cef0bf8c-0590-445c-ae6f-b019724166cd - - - - -] Failed to contact the endpoint at https://placement.openstack.maas:8778 for discovery. Fallback to using that endpoint as the base url.
2023-03-22 18:10:26.828 1208781 CRITICAL nova [req-cef0bf8c-0590-445c-ae6f-b019724166cd - - - - -] Unhandled error: openstack.exceptions.NotSupported: The placement service for keystone.openstack.maas:RegionOne exists but does not have any supported versions.

And the following error on the nova-compute node

2023-03-22 18:03:55.673 3837073 WARNING nova.conductor.api [req-b044d3a6-a656-407f-9018-20ecfcc1dee4 - - - - -] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor? Reattempting establishment of nova-conductor connection...: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 44661288325d486fb41cb9b81c310d71

keystone_error.log is empty

I hope someone can help me with this.

2

I found the root problem, for some reason placement nodes didn’t have new certificates , i reissue the certificates and the cloud is working properly now

3

I’m glad you found the problem. It’s disconcerting however that Placement was missing its certificates.

4

Yes is very disconcerting, I am not sure why it happened.

5

@pmatulis It seams it happen again placement does not have the new certificates, but this time I can not fix it as before .

6

placement is listed as related to vault

geoint@MAAS-01:~$ juju status --relations | grep vault
vault                             1.5.9    active       3  vault                      stable    54  no       Unit is ready (active: true, mlock: disabled)
vault-hacluster                            active       3  hacluster                  stable    81  no       Unit is ready and clustered
vault-mysql-router                8.0.36   active       3  mysql-router               stable    15  no       Unit is ready
vault/0                                active    idle   0/lxd/6  10.2.101.142    8200/tcp           Unit is ready (active: true, mlock: disabled)
  vault-hacluster/1                    active    idle            10.2.101.142                       Unit is ready and clustered
  vault-mysql-router/1                 active    idle            10.2.101.142                       Unit is ready
vault/1*                               active    idle   1/lxd/6  10.2.101.159    8200/tcp           Unit is ready (active: true, mlock: disabled)
  vault-hacluster/0*                   active    idle            10.2.101.159                       Unit is ready and clustered
  vault-mysql-router/0                 active    idle            10.2.101.159                       Unit is ready
vault/2                                active    idle   2/lxd/6  10.2.101.174    8200/tcp           Unit is ready (active: true, mlock: disabled)
  vault-hacluster/2                    active    idle            10.2.101.174                       Unit is ready and clustered
  vault-mysql-router/2*                active    idle            10.2.101.174                       Unit is ready
mysql-innodb-cluster:db-router              vault-mysql-router:db-router                 mysql-router                    regular      
vault-hacluster:ha                          vault:ha                                     hacluster                       subordinate  
vault-hacluster:hanode                      vault-hacluster:hanode                       hacluster                       peer         
vault-mysql-router:shared-db                vault:shared-db                              mysql-shared                    subordinate  
vault:certificates                          ceph-radosgw:certificates                    tls-certificates                regular      
vault:certificates                          cinder:certificates                          tls-certificates                regular      
vault:certificates                          glance-simplestreams-sync:certificates       tls-certificates                regular      
vault:certificates                          glance:certificates                          tls-certificates                regular      
vault:certificates                          keystone:certificates                        tls-certificates                regular      
vault:certificates                          mysql-innodb-cluster:certificates            tls-certificates                regular      
vault:certificates                          neutron-api-plugin-ovn:certificates          tls-certificates                regular      
vault:certificates                          neutron-api:certificates                     tls-certificates                regular      
vault:certificates                          nova-cloud-controller:certificates           tls-certificates                regular      
vault:certificates                          openstack-dashboard:certificates             tls-certificates                regular      
vault:certificates                          ovn-central:certificates                     tls-certificates                regular      
vault:certificates                          ovn-chassis:certificates                     tls-certificates                regular      
vault:certificates                          placement:certificates                       tls-certificates                regular      
vault:cluster                               vault:cluster                                vault-ha                        peer
7

it seams that placement is getting the messages form the vault but the files never changed

 juju debug-log --include placement
unit-placement-1: 15:35:24 INFO unit.placement/1.juju-log certificates:84: Reactive main running for hook certificates-relation-changed
unit-placement-2: 15:35:24 INFO unit.placement/2.juju-log certificates:84: Reactive main running for hook certificates-relation-changed
unit-placement-0: 15:35:24 INFO unit.placement/0.juju-log certificates:84: Reactive main running for hook certificates-relation-changed
unit-placement-2: 15:35:24 INFO unit.placement/2.juju-log certificates:84: Initializing Leadership Layer (is follower)
unit-placement-1: 15:35:24 INFO unit.placement/1.juju-log certificates:84: Initializing Leadership Layer (is leader)
unit-placement-0: 15:35:24 INFO unit.placement/0.juju-log certificates:84: Initializing Leadership Layer (is follower)
unit-placement-0: 15:35:24 INFO unit.placement/0.juju-log certificates:84: Invoking reactive handler: reactive/placement_handlers.py:37:render_config
unit-placement-1: 15:35:24 INFO unit.placement/1.juju-log certificates:84: Invoking reactive handler: reactive/layer_openstack.py:126:default_request_certificates
unit-placement-2: 15:35:24 INFO unit.placement/2.juju-log certificates:84: Invoking reactive handler: reactive/placement_handlers.py:37:render_config
unit-placement-0: 15:35:24 INFO unit.placement/0.juju-log certificates:84: Making dir /etc/apache2/ssl/placement root:root 555
unit-placement-2: 15:35:24 INFO unit.placement/2.juju-log certificates:84: Making dir /etc/apache2/ssl/placement root:root 555
unit-placement-0: 15:35:24 INFO unit.placement/0.juju-log certificates:84: Making dir /etc/apache2/ssl/placement root:root 555
unit-placement-2: 15:35:24 INFO unit.placement/2.juju-log certificates:84: Making dir /etc/apache2/ssl/placement root:root 555
unit-placement-1: 15:35:24 INFO unit.placement/1.juju-log certificates:84: Invoking reactive handler: reactive/layer_openstack_api.py:20:default_setup_database
unit-placement-1: 15:35:24 INFO unit.placement/1.juju-log certificates:84: Invoking reactive handler: reactive/layer_openstack_api.py:37:default_setup_endpoint_connection
unit-placement-1: 15:35:24 INFO unit.placement/1.juju-log certificates:84: Invoking reactive handler: reactive/layer_openstack_api.py:54:default_update_peers
unit-placement-1: 15:35:25 INFO unit.placement/1.juju-log certificates:84: Invoking reactive handler: reactive/placement_handlers.py:37:render_config
unit-placement-1: 15:35:25 INFO unit.placement/1.juju-log certificates:84: Making dir /etc/apache2/ssl/placement root:root 555
unit-placement-1: 15:35:25 INFO unit.placement/1.juju-log certificates:84: Making dir /etc/apache2/ssl/placement root:root 555
unit-placement-1: 15:35:27 WARNING unit.placement/1.juju-log certificates:84: Not adding haproxy listen stanza for placement-api_int port is already in use
unit-placement-1: 15:35:27 WARNING unit.placement/1.juju-log certificates:84: Not adding haproxy listen stanza for placement-api_public port is already in use
unit-placement-1: 15:35:27 INFO unit.placement/1.juju-log certificates:84: Invoking reactive handler: reactive/placement_handlers.py:80:cluster_connected
unit-placement-1: 15:35:27 INFO unit.placement/1.juju-log certificates:84: Invoking reactive handler: hooks/relations/placement/provides.py:26:joined:placement
unit-placement-1: 15:35:27 INFO unit.placement/1.juju-log certificates:84: Invoking reactive handler: hooks/relations/tls-certificates/requires.py:79:joined:certificates
unit-placement-0: 15:35:27 WARNING unit.placement/0.juju-log certificates:84: Not adding haproxy listen stanza for placement-api_int port is already in use
unit-placement-0: 15:35:27 WARNING unit.placement/0.juju-log certificates:84: Not adding haproxy listen stanza for placement-api_public port is already in use
unit-placement-2: 15:35:27 WARNING unit.placement/2.juju-log certificates:84: Not adding haproxy listen stanza for placement-api_int port is already in use
unit-placement-2: 15:35:27 WARNING unit.placement/2.juju-log certificates:84: Not adding haproxy listen stanza for placement-api_public port is already in use
unit-placement-0: 15:35:28 INFO unit.placement/0.juju-log certificates:84: Invoking reactive handler: reactive/placement_handlers.py:80:cluster_connected
unit-placement-0: 15:35:28 INFO unit.placement/0.juju-log certificates:84: Invoking reactive handler: reactive/layer_openstack.py:126:default_request_certificates
unit-placement-2: 15:35:28 INFO unit.placement/2.juju-log certificates:84: Invoking reactive handler: reactive/placement_handlers.py:80:cluster_connected
unit-placement-2: 15:35:28 INFO unit.placement/2.juju-log certificates:84: Invoking reactive handler: reactive/layer_openstack_api.py:20:default_setup_database
unit-placement-2: 15:35:28 INFO unit.placement/2.juju-log certificates:84: Invoking reactive handler: reactive/layer_openstack_api.py:37:default_setup_endpoint_connection
unit-placement-2: 15:35:28 INFO unit.placement/2.juju-log certificates:84: Invoking reactive handler: reactive/layer_openstack_api.py:54:default_update_peers
unit-placement-0: 15:35:28 INFO unit.placement/0.juju-log certificates:84: Invoking reactive handler: reactive/layer_openstack_api.py:20:default_setup_database
unit-placement-2: 15:35:28 INFO unit.placement/2.juju-log certificates:84: Invoking reactive handler: reactive/layer_openstack.py:126:default_request_certificates
unit-placement-0: 15:35:28 INFO unit.placement/0.juju-log certificates:84: Invoking reactive handler: reactive/layer_openstack_api.py:37:default_setup_endpoint_connection
unit-placement-0: 15:35:28 INFO unit.placement/0.juju-log certificates:84: Invoking reactive handler: reactive/layer_openstack_api.py:54:default_update_peers
unit-placement-0: 15:35:28 INFO unit.placement/0.juju-log certificates:84: Invoking reactive handler: hooks/relations/placement/provides.py:26:joined:placement
unit-placement-0: 15:35:28 INFO unit.placement/0.juju-log certificates:84: Invoking reactive handler: hooks/relations/tls-certificates/requires.py:79:joined:certificates
unit-placement-2: 15:35:28 INFO unit.placement/2.juju-log certificates:84: Invoking reactive handler: hooks/relations/placement/provides.py:26:joined:placement
unit-placement-2: 15:35:28 INFO unit.placement/2.juju-log certificates:84: Invoking reactive handler: hooks/relations/tls-certificates/requires.py:79:joined:certificates
unit-placement-1: 15:35:28 INFO juju.worker.uniter.operation ran "certificates-relation-changed" hook (via explicit, bespoke hook script)
unit-placement-0: 15:35:29 INFO juju.worker.uniter.operation ran "certificates-relation-changed" hook (via explicit, bespoke hook script)
unit-placement-2: 15:35:29 INFO juju.worker.uniter.operation ran "certificates-relation-changed" hook (via explicit, bespoke hook script)


ubuntu@juju-5025f7-0-lxd-5:/etc/apache2/ssl/placement$ ls -laht
total 24K
-rw-r----- 1 root placement 1.7K Mar 23  2023 key_placement.openstack.maas
-rw-r----- 1 root placement 1.5K Mar 23  2023 cert_placement.openstack.maas
-rw-r----- 1 root placement 1.7K Mar 23  2023 key_juju-5025f7-0-lxd-5.maas
-rw-r----- 1 root placement 1.5K Mar 23  2023 cert_juju-5025f7-0-lxd-5.maas
dr-xr-xr-x 2 root root      4.0K Mar  3  2022 .
lrwxrwxrwx 1 root root        56 Mar  3  2022 cert_10.2.101.146 -> /etc/apache2/ssl/placement/cert_placement.openstack.maas
lrwxrwxrwx 1 root root        56 Mar  3  2022 cert_10.2.101.207 -> /etc/apache2/ssl/placement/cert_placement.openstack.maas
lrwxrwxrwx 1 root root        55 Mar  3  2022 key_10.2.101.146 -> /etc/apache2/ssl/placement/key_placement.openstack.maas
lrwxrwxrwx 1 root root        55 Mar  3  2022 key_10.2.101.207 -> /etc/apache2/ssl/placement/key_placement.openstack.maas
drwxr-xr-x 3 root root      4.0K Mar  3  2022 ..

other services got the certificates and are working fine


ubuntu@juju-5025f7-3-lxd-1:/etc/apache2/ssl/glance$ ls -ahlt
total 24K
-rw-r----- 1 root root 1.7K Mar 25 21:35 key_juju-5025f7-3-lxd-1.maas
-rw-r----- 1 root root 1.5K Mar 25 21:35 cert_juju-5025f7-3-lxd-1.maas
-rw-r----- 1 root root 1.7K Mar 25 21:35 key_glance.openstack.maas
-rw-r----- 1 root root 1.5K Mar 25 21:35 cert_glance.openstack.maas
dr-xr-xr-x 2 root root 4.0K Mar  3  2022 .
lrwxrwxrwx 1 root root   50 Mar  3  2022 cert_10.2.101.150 -> /etc/apache2/ssl/glance/cert_glance.openstack.maas
lrwxrwxrwx 1 root root   50 Mar  3  2022 cert_10.2.101.202 -> /etc/apache2/ssl/glance/cert_glance.openstack.maas
lrwxrwxrwx 1 root root   49 Mar  3  2022 key_10.2.101.150 -> /etc/apache2/ssl/glance/key_glance.openstack.maas
lrwxrwxrwx 1 root root   49 Mar  3  2022 key_10.2.101.202 -> /etc/apache2/ssl/glance/key_glance.openstack.maas
8

I was able solve this problem by removing the relation and creating it again

juju remove-relation  placement:certificates vault:certificates
juju add-relation placement:certificates vault:certificates

Placement now has the new certificate but my cloud still has problems that didt had before the certificate issues.

mchc@ubuntu:/data/git/openstack-cfg/client$ openstack server list --all-projects
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'neutronclient.common.exceptions.ServiceUnavailable'> (HTTP 500) (Request-ID: req-6f1ca795-bf57-448b-8c5a-be404415a2c0)