Kubeflow 1.7 - Dex auth upgrade hook failed

charm
#1

I had a working Kubeflow 1.7 model until last night. I tried logging into Kubeflow and was met with a blank page with text “no healthy upstream.”

juju status reveals that dex-auth is stuck waiting with message “installing agent”, and the unit message says hook failed: "upgrade-charm". I can’t seem to get any useful log output to debug, and removing dex-auth doesn’t help (it arrives at the same error upon trying to reinstall).

Is there any known bug I should look out for? I’ve not changed any config beyond the basics outlined in the guide.

#2

I managed to fish out some logs:

failed to initialize server: server: Failed to open connector gitlab: failed to open connector: failed to create connector gitlab: failed to get provider: oidc: failed to decode provider discovery object: expected Content-Type = application/json, got "text/html": invalid character '<' looking for beginning of value 2023-04-11T11:27:26Z ERROR cannot start service: exited quickly with code 2

I removed my auth configuration with juju config dex-auth connectors="" then reapplied it with juju config dex-auth connectors="$(cat somefile.json)" and dex-auth came back to normal.

What I don’t understand at all is what caused the behavior. Is this going to happen every few days now? In any case, the immediate problem seems to be solved.