Juju vault also causing problems when deploying openstack/base on maas and charmed-kubernetes

I have deployed openstack/base on MaaS as indicated here. After I tried to deploy charmed-kubernetes with an openstack-integrator and vault overlay, I cannot perform openstackclient commands on the maas node and the images uploaded to the dashboard are not recognized, that means, the ubuntu charms cannot be deployed. When I do, for example,

openstack catalog list

I get

Failed to discover available identity versions when contacting https://keystone_ip:5000/v3. Attempting to parse version from URL.
SSL exception connecting to https://keystone_ip:5000/v3/auth/tokens: HTTPSConnectionPool(host=‘keystone_ip’, port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)’)))

However, when I ssh into the keystone container, there is a keystone_juju_ca_cert.crt which has as

Issuer: CN = Vault Root Certificate Authority (charm-pki-local)

and as

Subject: CN = Vault Root Certificate Authority (charm-pki-local)

I have also tried to reissue the certificates and refresh the secrets through actions in the vault application, but to no avail.

Can somobody help me here ? By this point, this is already the second bigger problem I have with certificate management using vault. Is there a more comprehensive guide with more detailed information ?

Hi @paulrusu. Sorry to hear that you’re having trouble. Can any of the @openstack-charmers offer any recommendations?

To me, it looks like your client doesn’t know about the CA that Vault is using. Have you added Vault’s CA to the machine where you’re trying to run openstack catalog list?

The CA can be retrieved with juju run-action vault/leader --wait get-root-ca and should then be added into your environment with something like export OS_CACERT=/path/to/that/CA.crt

Thanks, that was the cause. Maybe this is also the origin of the problem I had with the private docker registry, but I doubt it. I will try that one, too, but now that I am trying to deploy openstack/base on maas, I am having another type of certificate problem that I will post a question to, too.

Although I have initialized vault, when I run this command
juju run-action vault/leader --wait get-root-ca

it does not get any crt

UnitId: vault/0
id: “76”
Stderr: |
All snaps up to date.
Stdout: |
output: None
status: completed
completed: 2020-12-09 12:46:15 +0000 UTC
enqueued: 2020-12-09 12:46:05 +0000 UTC
started: 2020-12-09 12:46:07 +0000 UTC

I am facing this integration issue, it used to be ok, but suddently integration failed for new deployment.

unit-openstack-integrator-0: 10:27:41 DEBUG unit.openstack-integrator/0.config-changed SSL exception connecting to HTTPSConnectionPool(host=‘’, port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by SSLError(SSLError(“unable to load trusted certificates: Error([(‘system library’, ‘fopen’, ‘Permission denied’), (‘BIO routines’, ‘BIO_new_file’, ‘system lib’), (‘x509 certificate routines’, ‘X509_load_cert_crl_file’, ‘system lib’)],)”,),))

When i ssh to unit openstack-integrator, ca.crt exists
ubuntu@juju-b1c354-shared-k8s-9:~$ ls -lah /etc/openstack-integrator/
total 12K
drwxr-xr-x 2 root root 4.0K Mar 21 10:20 .
drwxr-xr-x 94 root root 4.0K Mar 21 10:20 …
-rw-r–r-- 1 root root 1.3K Mar 21 10:48 ca.crt

@arakattack, it looks like you’re running into this fairly new bug

Hi Chris, Thanks

My current workaround is to add apparmor manually on openstack-integrator container.

It Maybe like different between openstack-base #73 and openstack-base-70

openstack-base-70 's ca.crt in /tmp
openstack-base-73 's ca.crt in ~/snap/openstackclients/common/root-ca.crt

Here we go…

juju ssh openstack-integrator/0
sudo su
vi /var/lib/snapd/apparmor/profiles/snap.openstackclients.openstack
put this bellow 2 lines

/etc/openstack-integrator/* rw,

/proc/*/mounts rw,

and then
apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.openstackclients.openstack
juju resolve openstack-integrator/0

1 Like