Calico SNAT issues K8s 1.19 and 1.20 not with 1.18

juju
1

I’ve deployed multiple K8s clusters with Juju. When deploying 1.19 or 1.20 with Calico, a pod is not able to access an external IP through SNAT. Calico is configured to allow SNAT. When I build a 1.18 K8s cluster, the pods are able to communicate with external devices through SNAT.

Here are some of the errors within the Calico logs:
W0222 08:16:12.504357 1 reflector.go:302] pkg/mod/k8s.io/client-go@v12.0.0+incompatible/tools/cache/reflector.go:98: watch of *v1.Pod ended with: too old resource version: 356612 (565834)

233: Failed to reach apiserver error=
I0222 08:16:10.065125 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: []

2

I have to just confirm the issue you describe, same in our setup. Pods are reachable through MetalLB exposed service without issues, but any attempt to reach out any IP/hostname outside the cluster from pod itself fails (like curl, ping, traceroute, …). It seems despite that there are routes set by Calico on worker node that somehow worker/Calico is unable to route traffic out. More details here: https://calicousers.slack.com/archives/CPEPF833L/p1635509784033600