Replacing EasyRSA with Vault on Charmed K8s

charm
,
1

Hi,

I’ve been trying to replace EasyRSA with Vault on Charmed Kubernetes v1.19 following the guide - Transitioning an existing cluster from EasyRSA to Vault .
I’ve configured Vault as an Intermediate CA signed by a self signed Root CA.

  1. After removing the EasyRSA charm, and restarting the kubeapi-lb, it’s in error state with the following error log:
2021-08-29 16:33:58 WARNING start Traceback (most recent call last):
2021-08-29 16:33:58 WARNING start   File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/charm/hooks/start", line 22, in <module>
2021-08-29 16:33:58 WARNING start     main()
2021-08-29 16:33:58 WARNING start   File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
2021-08-29 16:33:58 WARNING start     bus.dispatch(restricted=restricted_mode)
2021-08-29 16:33:58 WARNING start   File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
2021-08-29 16:33:58 WARNING start     _invoke(other_handlers)
2021-08-29 16:33:58 WARNING start   File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
2021-08-29 16:33:58 WARNING start     handler.invoke()
2021-08-29 16:33:58 WARNING start   File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
2021-08-29 16:33:58 WARNING start     self._action(*args)
2021-08-29 16:33:58 WARNING start   File "/var/lib/juju/agents/unit-kubeapi-load-balancer-0/charm/reactive/tls_client.py", line 51, in store_server
2021-08-29 16:33:58 WARNING start     server_cert = server_cert + '\n' + chain
2021-08-29 16:33:58 WARNING start TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
2021-08-29 16:33:58 ERROR juju.worker.uniter.operation runhook.go:139 hook "start" (via explicit, bespoke hook script) failed: exit status 1
2021-08-29 16:33:58 INFO juju.worker.uniter resolver.go:143 awaiting error resolution for "start" hook

When I print the server_cert, server_key and chain from store_server(), I get empty values for the first two:

2021-08-29 16:59:21 INFO juju-log Server Certificate: None

2021-08-29 16:59:21 INFO juju-log Server Key: None

2021-08-29 17:00:28 INFO juju-log Chain Certificate: -----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----

  1. Also, the client returns the unknown CA error although I’ve installed the CA cert on all nodes:
$ kubectl get all
Unable to connect to the server: x509: certificate signed by unknown authority

Would be grateful for any suggestions on how I should resolve this?

Thanks & Regards,
Sana

2

It looks like this was filed a little while back in tls-client layer #22 and is due to a flag not being properly cleaned up in the tls-certificates interface layer. The proper fix is for the interface layer to be converted to implement the manage_flags method (or, even better, converted to the Operator Framework using the ops-reactive-interface helper library to maintain compatibility, but that’s a more significant undertaking).

A short-term workaround would be to clear the offending flag manually with something like:

$ juju run --unit kubeapi-load-balancer/0 -- charms.reactive clear_flag certificates.server.cert.available
$ juju resolved kubeapi-load-balancer/0

That should allow the removal to proceed, after which you should be able to add the Vault relation.

It might also be possible to add Vault prior to removing EasyRSA to avoid the bug, but I haven’t tested whether that will not have its own issues, since the charms aren’t really designed to support multiple CAs / cert providers.

1 Like