kubeflow dashboard fails to come up and displays "Internal server error failed to connect to the database"

getting started
, , ,
1

I’m setting up a two node on premise k8 cluster following this guide charmed-kubeflow.

Environment:

OS : Ubuntu 20.04

RAM: 64 GB

DISK: 1 TB HDD

CORES: 24

Day 1 - I was able to login to kubeflow dashboard as stated in the instructions[once] and access various components like notebooks, pipelines etc.

Day 2 - The machines were in ON a state throught and no reboot or ip-address has changed. But for some reason I’m not able to access the kubeflow dashboard again. The dex auth fails and throws error “Internal server error failed to connect to the database”. I have checked for juju status , microk8s status and inspect and attaching the outputs of these command for reference.

dex-error
dex-error1848×968 48.7 KB

Any help is appreciated.

2

Hello @sriram-sriram . Would you mind sharing also the results of juju status and microK8s status?

3

Hi, @munteanuandreea . Since I’m a new user I was not allowed to upload more attachments. I have clubbed juju status and microk8s status into one merged file. Kindly, take a look.

juju-mk8-status
juju-mk8-status650×2447 425 KB

Thanks.

4

Hey @sriram-sriram, I can try to help but looks like we’ll need to do some more technical deep dive here. Could you open an issue to continue this discussion on GH https://github.com/canonical/dex-auth-operator/issues?

With a first look it seems Dex is returning some 500 error and we need to inspect this to understand.

In your issue can you also give tell us the request from your browser’s network tab that returned an error (most probably a 5xx erro)? If you are not sure how to do this we can continue this on the issue

5

Hello @kimwnasptd , the issue got resolved after cleanup and re-installation.

1 Like