502 Bad Gateway

We have hit the following issue with k8s deployed with juju:

$ juju status
Model    Controller      Cloud/Region         Version  SLA          Timestamp
default  my1-controller  localhost/localhost  2.9.32   unsupported  09:27:34Z

App                    Version   Status  Scale  Charm                  Channel   Rev  Exposed  Message
containerd             go1.13.8  active      2  containerd             stable    178  no       Container runtime available
easyrsa                3.0.1     active      1  easyrsa                stable    420  no       Certificate Authority connected.
etcd                   3.4.5     active      3  etcd                   stable    634  no       Healthy with 3 known peers
flannel                0.11.0    active      2  flannel                stable    597  no       Flannel subnet
kubeapi-load-balancer  1.18.0    active      2  kubeapi-load-balancer  stable    844  yes      NGINX is ready
kubernetes-master      1.22.15   active      1  kubernetes-master      stable   1078  no       Kubernetes master running.
kubernetes-worker      1.22.15   active      1  kubernetes-worker      stable    816  yes      Kubernetes worker running.

Unit                      Workload  Agent  Machine  Public address  Ports             Message
easyrsa/0*                active    idle   0                      Certificate Authority connected.
etcd/0                    active    idle   1    2379/tcp          Healthy with 3 known peers
etcd/1                    active    idle   2    2379/tcp          Healthy with 3 known peers
etcd/2*                   active    idle   3    2379/tcp          Healthy with 3 known peers
kubeapi-load-balancer/1   active    idle   13                      NGINX is ready
kubeapi-load-balancer/2*  active    idle   14    443/tcp,6443/tcp  Loadbalancer ready.
kubernetes-master/0*      active    idle   5    6443/tcp          Kubernetes master running.
  containerd/4            active    idle                        Container runtime available
  flannel/4               active    idle                        Flannel subnet
kubernetes-worker/0*      active    idle   7    80/tcp,443/tcp    Kubernetes worker running.
  containerd/1*           active    idle                        Container runtime available
  flannel/1*              active    idle                        Flannel subnet

Machine  State    Address       Inst id         Series  AZ  Message
0        started  juju-e1704d-0   focal       Running
1        started  juju-e1704d-1   focal       Running
2        started  juju-e1704d-2   focal       Running
3        started  juju-e1704d-3   focal       Running
5        started  juju-e1704d-5   focal       Running
7        started  juju-e1704d-7   focal       Running
13       started  juju-e1704d-13  focal       Running
14       started  juju-e1704d-14  focal       Running

$ k get pods -n test
Error from server (InternalError): an error on the server ("<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.18.0 (Ubuntu)</center>\r\n</body>\r\n</html>") has prevented the request from succeeding

But if I run exactly the same command (k get pods -n test) a few seconds later it works correctly.

Can anybody tell me what is wrong ?

Hi! To clarify, does the command fail every now and then, or just once, the first time you call it, as soon as charmed k8s is done deploying? Is the issue encountered in an automated script, or what is exactly the repro steps for this behaviour?

It could be that you’re trying to hit the k8s api ‘too soon’, when it’s not ready yet, but I’m no expert on this (I can find someone who is though :slight_smile: )

This command fails every now and then.
The k8s and api load balancer have been deployed long time ago.

no idea, but I asked to some colleagues. Someone will get back to you :slight_smile: