Can't access kubernetes dashboard launched from juju on aws

brightlee6 · 28 May 2020 22:45

I setup a kubernetes cluster in aws using juju following this link:

To access kubernetes dashboard, I ran following on one ec2 instance(with juju and kubectl setup):
kubectl proxy --address 0.0.0.0 --accept-hosts ‘.*’

Then I access the dashboard using:
http://:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

After I provided k8s config file, but got following error with never login:

Please provide help on how to access kubernetes dashboard.

Thanks.

brightlee6 · 28 May 2020 22:47

Here is the msg from “kubectl logs kubernetes-dashboard-7cf54d76b5-qj7fb -n kubernetes-dashboard -f”, for your reference.

camille.rodriguez1 · 29 May 2020 13:44

Maybe try a different browser, or a different method of authentication? That’s odd, I have never seen that error before.

kos.tsakalozos · 29 May 2020 13:51

@brightlee6 could you try

kubectl port-forward -n kube-system service/kubernetes-dashboard 10443:443 --address 0.0.0.0

brightlee6 · 29 May 2020 18:51

Same error on both Chrome and Firefox.

Here is screen shot from Chrome:

What other authentication methods I can try here?

Thanks.

cory_fu · 3 June 2020 13:47

It looks like this error means that the Dashboard doesn’t understand the auth method specified in the Kubeconfig file. It might be helpful to post the contents (with sensitive info redacted) of that file so that we can see what went wrong.

The failure, though, should be handled more gracefully and surfaced, which appears to have been fixed so hopefully that will be resolved soon and we’ll get a more useful error surfaced if something like this happens.

brightlee6 · 4 June 2020 15:55

Here is the config file, which is very simple. Just includes username and password. I tried the username and password login and got same errors.

Thanks.

kwmonroe · 4 June 2020 16:28

Can you also share the details of the dashboard container from:

kubectl describe po -n kubernetes-dashboard -l k8s-app=kubernetes-dashboard

It’d be especially useful to see the “Image” and “Args” used for the dashboard container.

brightlee6 · 5 June 2020 16:19

Here is the screen shot from the output
kubectl describe po -n kubernetes-dashboard -l k8s-app=kubernetes-dashboard

Thanks.

brightlee6 · 5 June 2020 16:21

Here is the second half image:

By the way, I also tried using token to login, but got same error:
juju config kubernetes-master dashboard-auth==token

Thanks.

kwmonroe · 5 June 2020 19:15

Hm, that seems ok. I have a couple more things to try. First, are you using keystone in this deployment, or is this just vanilla charmed-kubernetes deployed to aws? FWIW, you would have had to manually deploy keystone and add a relation; if you didn’t do that, you’re not using it.

Assuming no keystone, set the dashboard auth back to basic (auto should also work, but let’s be explicit just in case):

juju config kubernetes-master dashboard-auth='basic'

And let’s verify the admin passwords (output is just for reference; don’t share your version here):

$ juju run --application kubernetes-master -- grep admin /root/cdk/basic_auth.csv
- Stdout: |
    7lAk3lItQvu5l710awgHDLmBLV5KLrAR,admin,admin,system:masters
  UnitId: kubernetes-master/0
- Stdout: |
    7lAk3lItQvu5l710awgHDLmBLV5KLrAR,admin,admin,system:masters
  UnitId: kubernetes-master/1

Again, don’t share your output; I just want you to double check that the first field from that admin line matches for all k8s-master units and that it matches the password that you have in your kube config file. That first csv field value should allow you to login using Basic auth with admin as the username and <value> as the password.

If all the passwords match, let’s try hitting the dashboard directly instead of via your kubectl proxy machine. Find the public ip of the kubeapi-load-balancer:

$ juju status kubeapi-load-balancer
...
App                    Version  Status  Scale  Charm                  Store       Rev  OS      Notes
kubeapi-load-balancer  1.14.0   active      1  kubeapi-load-balancer  jujucharms  729  ubuntu  exposed

Unit                      Workload  Agent  Machine  Public address  Ports    Message
kubeapi-load-balancer/0*  active    idle   4        54.177.155.234  443/tcp  Loadbalancer ready.

Two things to note from above: (a) make sure the kubeapi-load-balancer says exposed in the Notes column, and (b) use the Public address IP to access the dashboard. From above, I would navigate to:

https://54.177.155.234/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login

Substitute your public address in place of mine and see if you can login with either Kubeconfig or Basic auth mechanisms.

To recap:

are you using keystone for auth?
explicitly configure k8s-master to set dashboard-auth='basic'
verify the admin password matches across all k8s-master units and the password from your kubeconfig file
try to connect to the kubeapi-load-balancer directly without going through the machine that you’re using to run kubectl proxy

If any of those result in success, we can hopefully figure out where things went wrong. Thanks for your patience and willingness to debug here!

brightlee6 · 8 June 2020 17:53

Thanks for these details.

Yes I didn’t use keystone. So I followed your steps, set dashboard-auth=‘basic’; verified the admin password matches, and connected to the kubeapi-load-balancer directly.

Great news! I can access the dashboard using kubeapi-load-balancer, after three login.

Thanks.