Traefik-k8s docs - Troubleshooting reachability

charm
,
1

After relating traefik to your charm, you may encounter reachability errors such as:

  • Internal server error
  • 404 page not found

Checklist

  • Your charm either uses open-port1,2 or applies a kubernetes service patch for exposing the workload’s port.
  • Your workload is not bound to a particular IP address, i.e. it is listening on “0.0.0.0”.
  • You either pass strip_prefix=True to the ingress library, or your workload’s pebble command includes the path prefix generated by traefik (/model_name-app_name) so your app is serving correctly.
  • The SANs in your workload’s certificate match the url in traefik’s config.
  • Traefik trusts the same CA that signed your workload’s certificate.
  • Inspect logs.

Workload is correctly exposed

juju ssh into the charm container and run ss -tulpn. Find your app’s port in the list and make sure it is not bound to a particular IP, e.g.

Local Address:Port
            *:3200
            *:9096

Also make sure kubernetes has your ports listed as ClusterIP, for example:

$ kubectl -n welcome-k8s get svc                       
NAME    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                   AGE
tempo   ClusterIP   10.152.183.237   <none>        3200/TCP,4317/TCP,4318/TCP,9411/TCP,14268/TCP,14250/TCP   30m

Ingress path prefix is handled correctly

By default, traefik uses deterministic paths to all proxied endpoints, such as http://<traefik ip>/model_name-app_name.

This means that you need to tell your workload about it (for example alertmanager takes a --web.external-url cli arg) or tell traefik to strip prefix:

        self.ingress = IngressPerAppRequirer(
            self,
            "ingress",
            port=self._port,
            scheme=lambda: "https" if self.server_cert.cert else "http",
            redirect_https=True,
            strip_prefix=True,
        )

Workload’s certificate has the correct SAN DNS

To avoid using insecureSkipVerify in traefik configuration, the url specified in the traefik config must match the DNS in the cert.

Shell into the traefik workload container to confirm:

$ juju ssh --container traefik trfk/0 bash

root@trfk-0:/# cat /opt/traefik/juju/juju_ingress_ingress_48_am.yaml | grep url
        - url: https://am-0.am-endpoints.welcome-k8s.svc.cluster.local:9093

root@trfk-0:/# echo | openssl s_client -showcerts -connect am-0.am-endpoints.welcome-k8s.svc.cluster.local:9093 | openssl x509 -text | grep DNS
                DNS:am-0.am-endpoints.welcome-k8s.svc.cluster.local

Traefik trusts the same CA that signed your workload’s certificate

To avoid using insecureSkipVerify in traefik configuration, traefik must trust the CA that signed the proxied app’s certificate.

Shell into the traefik workload container to confirm:

$ juju ssh --container traefik trfk/0 bash

root@trfk-0:/# cat /opt/traefik/juju/juju_ingress_ingress_48_am.yaml | grep -E "url|cert"
      - /opt/traefik/juju/ca.cert
        - url: https://am-0.am-endpoints.welcome-k8s.svc.cluster.local:9093

root@trfk-0:/# curl --capath /opt/traefik/juju \
    --cacert /opt/traefik/juju/ca.cert \
    https://am-0.am-endpoints.welcome-k8s.svc.cluster.local:9093/welcome-k8s-am/-/ready
OK

Inspect traefik’s log

Make sure traefik is running.

$ juju ssh --container traefik trfk/0 /charm/bin/pebble changes traefik
ID   Status  Spawn               Ready               Summary
1    Done    today at 13:20 UTC  today at 13:20 UTC  Replan service "traefik"

$ juju ssh --container traefik trfk/0 /charm/bin/pebble tasks 1        
Status  Spawn               Ready               Summary
Done    today at 13:20 UTC  today at 13:20 UTC  Start service "traefik"

Look at the traefik log as you add/remove relations:

kubectl -n welcome-k8s logs pods/trfk-0 -c traefik -f
1 Like