cross node pod communication (pod-pod) does not work.

We are using Charmed Kubernetes and cross node pod communication does not work. This cluster is installed on OpenStack cloud. I am using the provider network in OpenStack Could anybody shed light on this?

All security groups are allowed and the juju controllers and k8s nodes are on the same network.

10.121.83.0/24 node network 10.253.0.0/16 pod network

Calcio configuration is out of the box standard BGP nod-node mesh config nothing changed. Calcio and all other components has standard configs which comes with charmed k8s. The POD CIDR is changed from default to above.

Nodes:

Pods:

Ping source K8s control plane: 10.121.83.217 Destination Pod IP: 10.253.20.204

TCPDUMP on source

Route details on source

Ping Destination Worker node: 10.121.83.96 Destination Pod Ip: 10.253.20.204

TCPDUMP on Destination

Route details on Destination