Pulling image from private docker registry on host causes unknown certificate authority error although certificate installed

I have deployed kubernetes-core through juju on two ubuntu bionic host machines (microstack). On my controller node, there is a private docker registry with my web-application image that I want to deploy. However, I always get error

  Warning  Failed     12s (x2 over 26s)  kubelet, juju-50ab96-default-1  Failed to pull image "registry_ip:5001/frontend:latest": rpc error: code = Unknown desc = failed to pull and unpack image "registry_ip:5001/frontend:latest": failed to resolve reference "registry_ip:5001/frontend:latest": failed to do request: Head https://registry_ip:5001/v2/frontend/manifests/latest: x509: certificate signed by unknown authority

The registry is secured by a self-signed certificate, whose certificate authority cert is deployed to the juju client node as described in https://kubernetes.io/docs/concepts/cluster-administration/certificates/. Does anyone understand this behavior and know how to remedy it ?

Hi Paul, welcome

Just guessing, but it looks like there is a variable (“registry_ip”) that’s being sent as a string rather than interpreted.

Out of interest, do you know what steps you took to deploy kubernetes-core? My understanding is that there shouldn’t be a need to configure a private registry manually.

You should be able to run

juju run-action kubernetes-worker/leader registry

I believe that there’s also a private-registry charm, but I’ve read that its documentation is out-of-date.

The @k8s-charmers team can advise :slight_smile:

Thanks for the reply. The “registry_ip” is just a change I made before posting the error message here. Although it is not a critical environment and the IP-address is local, I don’t want to get used to bad habits.
I also tried the private-registry deployment as indicated here. However, this but displaces the problem as I can push images to that registry, only to then have the problem of having to push all images to that registry if I designate it as the default registry through juju config kubernetes-master image-registry=$REGISTRY. Thus, my question is rather about the fact why kubelet does not allow the certificate, although it is located in /usr/local/share/ca-certificates/kubernetes.crt ?

By the way, I am pretty new to juju, having previously worked with kubeadm kubernetes on vagrant. Therefore, I already have a fairly developed environment with web applications that I would now like to transfer as smoothly as possible to an openstack/kubernetes environment.

General tip on certificates/ssl/tls: Always check the time on the host before chasing other errors.

If your local time is significantly different than expected, you will get all kinds of problems.

Hi Paul, welcome

The issue is the Kubernetes node does not have the CA certificate for the Docker registry.

I am not sure how Kubernetes is being deploy in your situation. But one of two solutions exists and can possible be controlled and set through the Kubernetes installation method.

  1. Adding the CA to the host system trust store should help fix it. See:
    Ubuntu Manpage: update-ca-certificates - update /etc/ssl/certs and certificates.crt
    https://superuser.com/questions/437330/how-do-you-add-a-certificate-authority-ca-to-ubuntu

  2. Use insecure registries in Docker’s config.
    Test an insecure registry | Docker Documentation

Cheers
Tom

@paulrusu this sounds dangerously familiar to this bug fixed by this PR. If so, it’ll be included Charmed Kubernetes 1.18+ck1 (slated for release Real Soon – as in days, not weeks).

From the initial description, it looks like you have the api server set to ack your custom CA (with the --client-ca-file param). That’s great, but probably unnecessary. This setting would help if k8s-worker nodes were trying to connect to the apiserver with a cert that wasn’t signed by something that the k8s-master trusted. This isn’t your problem.

Your problem is not that the k8s-master (apiserver) doesn’t know about your CA – it’s that the container runtime on your k8s workers doesn’t know about that CA. The thing that actually deploys a container image is the containerd or dockerd runtime that lives on your workers (fyi, containerd is the default since charmed-kubernetes 1.14, but dockerd is supported as well).

If you can wait for 1.18+ck1, you’ll be able to juju upgrade-charm containerd and add your CA like this:

juju config containerd custom-registry-ca=$(base64 ./ca.crt)

If you can’t wait, you should be able to juju scp /path/to/your-ca.crt kubernetes-worker/n:/usr/local/share/ca-certificates followed by juju run --unit kubernetes-worker/n 'update-ca-certificates --fresh', where n is the unit number of your worker.

Either way, the worker that is ultimately responsible for the image pull needs to know about the CA that signed your registry cert. Telling the master (via an apiserver --param) is not enough.

2 Likes