Kubeflow Behind Istio Returns 403 / 503 at Ingress, Dex & OIDC Gatekeeper Behaving Differently Across Two Identical Installations

, ,
1

Hello everyone,

I am troubleshooting an odd behavior between two Kubeflow installations that were both deployed using Juju charms. One installation is on-premise and working correctly, while the second one (hosted on a partner’s VM) fails with 403 or 503 responses at the Istio ingress — even though both deployments appear to have same configuration.

Working Cluster (Local)

  • Kubeflow deployed via Juju charms
  • MetalLB provides an external IP I can access the GUI on the browser
    image
    image1912×64 76.1 KB
    Using the IP of the host machine and the 31605 port.

###Broken Cluster (Remote Fedora41 Server)

  • Same Kubeflow deployment via Juju
  • MetalLB is currently configured with only one IP — the same as the host IP

image
image1320×50 35.6 KB

  • Accessing via browser or curl returns:

curl -i http://192.168.1.12:31811

HTTP/1.1 403 Forbidden

server: istio-envoy

I also tried to do curl from a pod inside the same ns, but again I got the same error.

I don’t know how to proceed.

Any ideas?

Thank you !