Adding trusted Certificate Authority certificates (CA certs) to machines, including machines yet to be provisioned

juju
1

Situation

You are deploying Juju models behind the firewall that is managed by another team. As part of the site’s hardening procedure, all SSL connections are routed through a jump box. You need to deploy extra Certificate Authority certificates issued by the other team onto every machine managed by Juju.

This situation was faced by Juju user @zicklag this week:

I am trying to setup Juju for local development using the localhost cloud and I am behind an HTTPS SSL bump proxy. Because of the SSL bump I have to add the proxy’s HTTPS certificate to the system certificate authority, but I haven’t found a way to automatically do that when bootstrapping a Juju controller or when adding a new Juju machine to the localhost cloud.

Solution for machines yet to be provisioned

Copy the CA cert(s) written in PEM format into a YAML file, making sure to use the vertical bar (|) to specify a multi-line string. You must also include the nested keys (cloudinit-userdata, ca-certs and trusted) as provided here:

cloudinit-userdata: |
  ca-certs:
    trusted: 
    - |
      -----BEGIN CERTIFICATE-----
      PEM-FORMATTTED-CA-CERTIFICATE
      -----END CERTIFICATE-----
    - |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----

Once the YAML file is created, use the model-config command to ask Juju to include it with the cloud-init user data. This example assumes that you have saved your YAML-formatted file as cloudinit-userdata.yaml:

juju model-config cloudinit-userdata.yaml

Background

Juju uses cloud-init to configure machines from their base images once they have been provisioned. The cloudinit-userdata model configuration setting (“model config key”) allows you to tweak what happens to machines when they are created up via the “user data” feature.

From the cloud-init website:

Cloud images are operating system templates and every instance starts out as an identical clone of every other instance. It is the user data that gives every cloud instance its personality and cloud-init is the tool that applies user data to your instances automatically

Solution for machines already provisioned

Ensure that the CA cert is saved in the PEM format with the .crt extension.

Run juju scp to copy the CA cert to the machine that you wish to update. (This example uses machine 0):

juju scp trusted-cert.crt 0:/usr/local/share/ca-certificates

Use juju exec (previously called juju run) to execute update-ca-certificates on machine 0.

juju exec --machine 0 --  update-ca-certificates

Further Reading

1 Like
2

Little problem with the YAML you gave as an example. There needs to be a | after cloudinit-userdata because that field is actually a string that happens to be a YAML document itself:

cloudinit-userdata: |
  ca-certs:
    trusted: 
    - |
      -----BEGIN CERTIFICATE-----
      PEM-FORMATTTED-CA-CERTIFICATE
      -----END CERTIFICATE-----
    - |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----

Also any specific place you think it would be useful to have this example of setting up a localhost cloud behind a proxy with SSL bump:

https://gist.github.com/zicklag/62d80b0f88324dd4e9d75ae3d8fb52eb

3

Thanks! I will fix that now.

4

Is there a way I can do this in the microk8s juju environment?. This particular solution seem to focus on machine based or lxd based deployments. I have a microk8s based environment and have the same exact problem and tried the cloudinit file way but did not help.

5

I don’t believe we currently have a solution for injecting MITM certificates before the application or charms come up in those environments.

We could look to do something equivalent to the cloud-init-userdata (seed a script that can be run as part of the init container). Though for this particular use case, I sort of prefer if Juju actually grew the concept of initial certificates, and then it could be applied consistently in both cases.

1 Like
6

Thanks @jameinel. It helps, I will look at the possibilities to have it exempt from SSL Interception.