Unable to deploy charms behind corporate proxy

getting started
#1

Hi there!

I’m trying to deploy Juju bootstrapping over microk8s behind a corporate proxy. I’ve checked several threads but still I’m unable to fix “juju deploy”:

$ juju deploy kubeflow
ERROR resolving with preferred channel: Post "https://api.charmhub.io/v2/charms/refresh": x509: 
certificate signed by unknown authority

I’m on Ubuntu 22.04.2 LTS

I’ve followed several steps - picked from various topics in the forum. Can anyone help me figure this out?

I prepared a bootstrap.yaml with all my proxy vars AND several ca-certs (full corporate cert chain) as my proxy will tamper with SSL:

apt-http-proxy: "my.corporate.proxy:port"
apt-https-proxy: "my.corporate.proxy:port"
apt-no-proxy: "my.corporate.domain,local,localhost,::1,127.0.0.0/8,10.0.0.0/8,192.168.0.0/16,172.16.0.0/16"
juju-http-proxy: "my.corporate.proxy:port"
juju-https-proxy: "my.corporate.proxy:port"
juju-no-proxy: "my.corporate.domain,local,localhost,::1,127.0.0.0/8,10.0.0.0/8,192.168.0.0/16,172.16.0.0/16"
snap-http-proxy: "my.corporate.proxy:port"
snap-https-proxy: "my.corporate.proxy:port"
cloudinit-userdata: |
  ca_certs:
    trusted: 
    - |
     -----BEGIN CERTIFICATE-----
     -----END CERTIFICATE-----
    - |
     -----BEGIN CERTIFICATE-----
     -----END CERTIFICATE-----
    - |
     -----BEGIN CERTIFICATE-----
     -----END CERTIFICATE-----
    - |
     -----BEGIN CERTIFICATE-----
     -----END CERTIFICATE-----
    - |
     -----BEGIN CERTIFICATE-----
     -----END CERTIFICATE-----

I bootstrap apparently OK with:

$ juju bootstrap --config=bootstrap.yaml --model-default=bootstrap.yaml microk8s micro
Creating Juju controller "micro" on microk8s/localhost
Bootstrap to Kubernetes cluster identified as microk8s/localhost
Fetching Juju Dashboard 0.8.1
Creating k8s resources for controller "controller-micro"
Starting controller pod
Bootstrap agent now started
Contacting Juju controller at 10.152.183.63 to verify accessibility...
Bootstrap complete, controller "micro" is now available in namespace "controller-micro"
Now you can run
    juju add-model <model-name>
to create a new model to deploy k8s workloads.

Some additional context:

Snap proxies are properly set up via /etc/environment and /var/snap/microk8s/current/args/containerd-env:

HTTP_PROXY=my.corporate.proxy:port
HTTPS_PROXY=my.corporate.proxy:port
NO_PROXY=my.corporate.domain,local,localhost,::1,127.0.0.0/8,10.0.0.0/8,192.168.0.0/16,172.16.0.0/16
http_proxy=my.corporate.proxy:port
https_proxy=my.corporate.proxy:port
no_proxy=my.corporate.domain,local,localhost,::1,127.0.0.0/8,10.0.0.0/8,192.168.0.0/16,172.16.0.0/16

Both microk8s and juju deployed and running just fine (snap install --classic microk8s && snap install --classic juju):

snap list | grep -e microk8s -e juju
juju                       2.9.42            22345  2.9/stable       canonical**    classic
microk8s                   v1.26.4           5219   1.26/stable      canonical**    classic

microk8s working just fine & with extra features enabled:

$ kubectl get all --all-namespaces
NAMESPACE            NAME                                            READY   STATUS    RESTARTS      AGE
minio-operator       pod/minio-operator-67dcf6dd7c-xn7fl             0/1     Pending   0             109m
kube-system          pod/kubernetes-dashboard-dc96f9fc-bw56h         1/1     Running   2 (78m ago)   116m
kube-system          pod/dashboard-metrics-scraper-7bc864c59-4xxmz   1/1     Running   2 (78m ago)   116m
kube-system          pod/hostpath-provisioner-69cd9ff5b8-tx2w4       1/1     Running   2 (78m ago)   116m
istio-system         pod/istiod-558cdbcbff-7s8wx                     1/1     Running   1 (78m ago)   110m
minio-operator       pod/console-66c4b79fbd-zg26q                    1/1     Running   1 (78m ago)   109m
container-registry   pod/registry-77c7575667-q66vh                   1/1     Running   2 (78m ago)   116m
kube-system          pod/calico-kube-controllers-79568db7f8-l678k    1/1     Running   2 (78m ago)   125m
kube-system          pod/coredns-6f5f9b5d74-5fff8                    1/1     Running   2 (78m ago)   117m
kube-system          pod/calico-node-8vztk                           1/1     Running   2 (78m ago)   125m
kube-system          pod/metrics-server-6f754f88d-xgpng              1/1     Running   2 (78m ago)   116m
istio-system         pod/istio-ingressgateway-5f57c68988-qwpnc       1/1     Running   1 (78m ago)   110m
istio-system         pod/istio-egressgateway-6b957f5b7d-kvjpg        1/1     Running   1 (78m ago)   110m
minio-operator       pod/minio-operator-67dcf6dd7c-qz2nc             1/1     Running   4 (77m ago)   109m
minio-operator       pod/microk8s-ss-0-0                             1/1     Running   4 (77m ago)   109m

NAMESPACE            NAME                                TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                      AGE
default              service/kubernetes                  ClusterIP      10.152.183.1     <none>        443/TCP                                                                      126m
kube-system          service/kube-dns                    ClusterIP      10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP                                                       117m
container-registry   service/registry                    NodePort       10.152.183.227   <none>        5000:32000/TCP                                                               117m
kube-system          service/metrics-server              ClusterIP      10.152.183.49    <none>        443/TCP                                                                      117m
kube-system          service/kubernetes-dashboard        ClusterIP      10.152.183.17    <none>        443/TCP                                                                      117m
kube-system          service/dashboard-metrics-scraper   ClusterIP      10.152.183.152   <none>        8000/TCP                                                                     117m
istio-system         service/istiod                      ClusterIP      10.152.183.214   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP                                        111m
istio-system         service/istio-egressgateway         ClusterIP      10.152.183.128   <none>        80/TCP,443/TCP                                                               110m
istio-system         service/istio-ingressgateway        LoadBalancer   10.152.183.67    <pending>     15021:30010/TCP,80:31753/TCP,443:30371/TCP,31400:30318/TCP,15443:30661/TCP   110m
minio-operator       service/operator                    ClusterIP      10.152.183.220   <none>        4222/TCP,4221/TCP                                                            109m
minio-operator       service/console                     ClusterIP      10.152.183.104   <none>        9090/TCP,9443/TCP                                                            109m
minio-operator       service/minio                       ClusterIP      10.152.183.183   <none>        80/TCP                                                                       109m
minio-operator       service/microk8s-console            ClusterIP      10.152.183.234   <none>        9090/TCP                                                                     109m
minio-operator       service/microk8s-hl                 ClusterIP      None             <none>        9000/TCP                                                                     109m

NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/calico-node   1         1         1       1            1           kubernetes.io/os=linux   126m

NAMESPACE            NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
kube-system          deployment.apps/calico-kube-controllers     1/1     1            1           126m
kube-system          deployment.apps/coredns                     1/1     1            1           117m
kube-system          deployment.apps/dashboard-metrics-scraper   1/1     1            1           117m
kube-system          deployment.apps/metrics-server              1/1     1            1           117m
kube-system          deployment.apps/kubernetes-dashboard        1/1     1            1           117m
kube-system          deployment.apps/hostpath-provisioner        1/1     1            1           117m
container-registry   deployment.apps/registry                    1/1     1            1           117m
istio-system         deployment.apps/istiod                      1/1     1            1           111m
minio-operator       deployment.apps/console                     1/1     1            1           109m
istio-system         deployment.apps/istio-ingressgateway        1/1     1            1           110m
istio-system         deployment.apps/istio-egressgateway         1/1     1            1           110m
minio-operator       deployment.apps/minio-operator              1/2     2            1           109m

NAMESPACE            NAME                                                  DESIRED   CURRENT   READY   AGE
kube-system          replicaset.apps/calico-kube-controllers-79568db7f8    1         1         1       125m
kube-system          replicaset.apps/coredns-6f5f9b5d74                    1         1         1       117m
kube-system          replicaset.apps/dashboard-metrics-scraper-7bc864c59   1         1         1       116m
kube-system          replicaset.apps/metrics-server-6f754f88d              1         1         1       116m
kube-system          replicaset.apps/kubernetes-dashboard-dc96f9fc         1         1         1       116m
kube-system          replicaset.apps/hostpath-provisioner-69cd9ff5b8       1         1         1       116m
container-registry   replicaset.apps/registry-77c7575667                   1         1         1       116m
istio-system         replicaset.apps/istiod-558cdbcbff                     1         1         1       110m
minio-operator       replicaset.apps/console-66c4b79fbd                    1         1         1       109m
istio-system         replicaset.apps/istio-ingressgateway-5f57c68988       1         1         1       110m
istio-system         replicaset.apps/istio-egressgateway-6b957f5b7d        1         1         1       110m
minio-operator       replicaset.apps/minio-operator-67dcf6dd7c             2         2         1       109m

NAMESPACE        NAME                             READY   AGE
minio-operator   statefulset.apps/microk8s-ss-0   1/1     109m

Anyone can help me spot the problem here?

#2

Hi @alienmind,

Thank you for the question. I am going to make some assumptions in this post so please correct anything I may have got wrong.

When you are bootstrapping with that config you are modifying the configuration for the Juju model and subsequently all applications/units deployed into that model get the configuration.

Because of this when deploying an application to a new model (the model that is not the one running the controller). These values will need to be set based on the context of where the model is running from. In your case this would be Kubernetes.

Please see:

Regards tlm

#3

Thanks for getting back to me, Tim.

Your assumptions are correct.

The error is triggered while accessing the Charm Store. Which, according to the links above:

Required by the controller so that charms can be deployed on the machines.See [Deploying charms offline]

What I’m trying to do is make the controller trust an intermediate proxy that is intercepting SSL.

Therefore I need to add certain ca.crt entries to the controller. I’ve prepared a bootstrap.yaml file with the relevant entries (cloudinit-userdata.ca_certs.trusted - see above)

However the controller seems to be ignoring the entries when running juju deploy.

Looking at Juju | Command 'juju model-config' there seems to be options for http_proxy / https_proxy / no_proxy, but I don’t see any particular config options for adding a custom ca.crt which seems to be the problem when going through an intercepting proxy.

Is there any other workaround I could try?