I have a functional test/PoC openstack deployment based on ubuntu@22.04/stable, 2023.2/stable and ceph reef. All services running and I can deploy working instances with functional network etc.
I am attempting to configure Ceilometer and Gnocchi, and while the configuration appears correct, I get errors in Ceilometer about nonUnique project names in Keystone. The issue relates to the fact their are 2 projects called “services”, one in the Default domain and the other in “service_domain”.
- ceilometer-agent-notification.log
2024-09-11 10:49:02.754 798958 ERROR ceilometer.publisher.gnocchi [-] Failed to retrieve filtered project [services].: keystoneclient.exceptions.NoUniqueMatch: ClientException
2024-09-11 10:49:02.754 798958 ERROR ceilometer.publisher.gnocchi Traceback (most recent call last):
2024-09-11 10:49:02.754 798958 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python3/dist-packages/ceilometer/publisher/gnocchi.py", line 288, in gnocchi_project_id
2024-09-11 10:49:02.754 798958 ERROR ceilometer.publisher.gnocchi project = self._ks_client.projects.find(
2024-09-11 10:49:02.754 798958 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python3/dist-packages/keystoneclient/v3/projects.py", line 225, in find
2024-09-11 10:49:02.754 798958 ERROR ceilometer.publisher.gnocchi p = super(ProjectManager, self).find(**kwargs)
2024-09-11 10:49:02.754 798958 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python3/dist-packages/keystoneclient/base.py", line 86, in func
2024-09-11 10:49:02.754 798958 ERROR ceilometer.publisher.gnocchi return f(*args, **new_kwargs)
2024-09-11 10:49:02.754 798958 ERROR ceilometer.publisher.gnocchi File "/usr/lib/python3/dist-packages/keystoneclient/base.py", line 506, in find
2024-09-11 10:49:02.754 798958 ERROR ceilometer.publisher.gnocchi raise ksc_exceptions.NoUniqueMatch
2024-09-11 10:49:02.754 798958 ERROR ceilometer.publisher.gnocchi keystoneclient.exceptions.NoUniqueMatch: ClientException
2024-09-11 10:49:02.754 798958 ERROR ceilometer.publisher.gnocchi
2024-09-11 10:49:02.754 798958 ERROR ceilometer.pipeline.sample [-] Pipeline network_sink: Continue after error from publisher <ceilometer.publisher.gnocchi.GnocchiPublisher object at 0x7fde60103340>: keystoneclient.exceptions.NoUniqueMatch: ClientException
+----------------------------------+----------------------------+----------------------------------+-----------------------------------------------+---------+
| ID | Name | Domain ID | Description | Enabled |
+----------------------------------+----------------------------+----------------------------------+-----------------------------------------------+---------+
| ec5964382c5b422bb76bfaa46d69a983 | services | 5be35b1bbb5a41a5a60cef025ebf9f93 | Created by Juju | True |
| f17bbdea04344547bd31dece98dc787a | services | default | Created by Juju | True |
+----------------------------------+----------------------------+----------------------------------+-----------------------------------------------+---------+
+----------------------------------+----------------+---------+--------------------+
| ID | Name | Enabled | Description |
+----------------------------------+----------------+---------+--------------------+
| 5be35b1bbb5a41a5a60cef025ebf9f93 | service_domain | True | Created by Juju |
| default | Default | True | The default domain |
+----------------------------------+----------------+---------+--------------------+
This duplicate I believe was/is to support the v2 and v3 Keystone API’s with v3 introducing “domains”.
- ceilometer.conf
[service_credentials]
...
project_name = services
project_domain_name = service_domain
user_domain_name = service_domain
...
[keystone_authtoken]
...
project_domain_name = service_domain
user_domain_name = service_domain
project_name = services
...
- pipeline.yaml
sinks:
- name: cpu_sink
publishers:
- gnocchi://?filter_domain=service_domain&filter_project=services&archive_policy=fleio_policy
- name: network_sink
publishers:
- gnocchi://?filter_domain=service_domain&filter_project=services&archive_policy=fleio_policy
- name: object_storage_sink
publishers:
- gnocchi://?filter_domain=service_domain&filter_project=services&archive_policy=fleio_policy
- name: ceph_sink
publishers:
- gnocchi://?filter_domain=service_domain&filter_project=services&archive_policy=fleio_policy
...
^ Have attempted with and without filter_domain, but that does seem to be a valid filter in /usr/lib/python3/dist-packages/ceilometer/publisher/gnocchi.py
I did attempt to prove the issue was the duplicate, by deleting one of the services projects, prior to a full redeployment, and the metrics were then successful and above error gone. This however breaks everything, and I can no longer deploy/delete instances etc. It seems Keystone grants to both “services” projects when deploying and fails to deploy services when either version is deleted.
2024-09-10 17:01:10 INFO unit.keystone/2.juju-log server.go:325 Granting user 'neutron' role 'Admin' on tenant 'services' in domain 'default'
2024-09-10 17:01:12 INFO unit.keystone/2.juju-log server.go:325 Granting user 'neutron' role 'Admin' on tenant 'services' in domain 'service_domain'
2024-09-10 17:01:22 INFO unit.keystone/2.juju-log server.go:325 Granting user 'neutron' role 'Admin' on tenant 'services' in domain 'default'
2024-09-10 17:01:24 INFO unit.keystone/2.juju-log server.go:325 Granting user 'neutron' role 'Admin' on tenant 'services' in domain 'service_domain'
2024-09-10 17:01:33 INFO unit.keystone/2.juju-log server.go:325 Granting user 'neutron' role 'Admin' on tenant 'services' in domain 'default'
2024-09-10 17:01:35 INFO unit.keystone/2.juju-log server.go:325 Granting user 'neutron' role 'Admin' on tenant 'services' in domain 'service_domain'
2024-09-10 17:01:44 INFO unit.keystone/2.juju-log server.go:325 Granting user 'nova' role 'Admin' on tenant 'services' in domain 'default'
2024-09-10 17:01:46 INFO unit.keystone/2.juju-log server.go:325 Granting user 'nova' role 'Admin' on tenant 'services' in domain 'service_domain'
2024-09-10 17:01:55 INFO unit.keystone/2.juju-log server.go:325 Granting user 'nova' role 'Admin' on tenant 'services' in domain 'default'
2024-09-10 17:01:58 INFO unit.keystone/2.juju-log server.go:325 Granting user 'nova' role 'Admin' on tenant 'services' in domain 'service_domain'
2024-09-10 17:02:06 INFO unit.keystone/2.juju-log server.go:325 Granting user 'nova' role 'Admin' on tenant 'services' in domain 'default'
2024-09-10 17:02:08 INFO unit.keystone/2.juju-log server.go:325 Granting user 'nova' role 'Admin' on tenant 'services' in domain 'service_domain'
Any pointers in how I can fix this issue without deleting one of the services projects, which broke other services. Both seem to be required, yet the fact there are 2 projects called “services” is breaking ceilometer.