ERROR cannot log into controller

juju
, ,
1

I was using the admin account default credential for too long. I decided to create some users and passwords. Having done so, I can no longer log in:

routergod@juju:~$ juju login -u routergod
please enter password for routergod on openstack:
ERROR cannot log into controller "openstack": cannot get discharge from "https://192.168.200.233:17070/auth": failed to acquire macaroon after waiting: third party refused discharge: discharging macaroon: interaction required

Any clues please? Many thanks in advance!

2

Can you try logging out and then logging back in? Macaroons only have a 24h shelf life I believe:

juju logout -u routergod
juju login -u routergod
3

Sure! No cigar unfortunatelyā€¦

routergod@juju:~$ juju logout -u routergod
ERROR option provided but not defined: -u
routergod@juju:~$ juju logout
Logged out. You are no longer logged into any controllers.
routergod@juju:~$ juju login -u routergod
please enter password for routergod on openstack:
ERROR cannot log into controller "openstack": invalid entity name or password
routergod@juju:~$ juju login -u routergod
please enter password for routergod on openstack:
ERROR cannot log into controller "openstack": cannot get discharge from "https://192.168.200.233:17070/auth": failed to acquire macaroon after waiting: third party refused discharge: discharging macaroon: interaction required

I purposely used the wrong password the first attempt there to show that there is apparently some password check going on.

I had a poke around but canā€™t find any log of the event on the controller.

4

Is the controller setup to use an external identity provider?

Juju does do password checks, but it returns a macaroon to the client.

  • Is the ip address shown 192.168.200.233 an ip address of the controller? I think it should be because port 17070 is the default API port on the controller.
  • The third party refused discharge is a bit strange to me. Clearly interaction required is being returned from somewhere. I wonder if there are missing environment variables in the client terminal. However Iā€™m not sure what it would be looking for.

@uros-jovanovic or @cmars do either of you have more insight to this error?

5

Yes the controller is 192.168.200.233. The controller is not knowingly set to use external identity, can I check that somehow?

6

Still no resolution for this, can someone suggest debugging approach please?

7

I prodded someone earlier today. Juju uses macaroons for remembering the login details rather than storing the password locally.

The bakery (part of the macaroon internals) tries to call out interactively to a browser.

Could you try again with this option set?

-B, --no-browser-login  (= false)
    Do not use web browser for authentication

Hopefully this will address the issue.

8

Thank you but unfortunately I get the same result with -B added.

Apologies for also raising a bug for this - desperation on my part.

9

Donā€™t appologise for raising a bug. It looks like there may well be a bug in there.

Are you sure you have the password right? Iā€™ve not come across this edge case before. Are you able to log back in as admin?

10

I have a bit of a problem there in that I set the admin password but somehow neglected to save the Keepass file! I was planning to try using the instructions that were posted here recently to recover that, once I understand this login issue. Before I lost the password though, attempting to log in as admin produced the same macaroon error.

I am using the correct password for the routergod user. If I purposely use the wrong password I get a more straightforward error message;

ERROR cannot log into controller "openstack": invalid entity name or password

I also created a user for a colleague, indeed that was my first action. At that time I had not attempted to set the admin password. Whatever default token was in place for the admin user was still working at that point. It seems the authorization bit is somehow not working where a password is set.

11

When a controller is bootstrapped, the admin password is saved in the accounts.yaml file. If you logout and then login, this is removed and turned into a macroon (I think). However, you should be able to find it there.

$ cat ~/.local/share/juju/accounts.yaml 
controllers:
  test:
    user: admin
    password: bd2aca662af1c962934368dc90fa6533
    last-known-access: superuser
12

Ha! Yeah that got trashed already :roll_eyes: I only have controllers.yaml here now.

I followed Bui Haā€™s instruction to reset the admin password in the database, which appears to have worked in that I get the macaroon error when I log in as admin with the new password I set;

routergod@juju:~/.local/share/juju$ juju login -u admin
please enter password for admin on openstack:
ERROR cannot log into controller "openstack": cannot get discharge from "https://192.168.200.233:17070/auth": failed to acquire macaroon after waiting: third party refused discharge: discharging macaroon: interaction required
13

@routergod :confused:

By ā€œBui Haā€™s instructionā€, do you mean this tutorial?

14

@routergod I went looking for the bug as @cmars asked for debug output. Something is clearly not working here, and Iā€™d love to know what.

I take it you have ssh access to the controller machine since you were able to reset the admin password. Is that right?

If so, is there anything in /var/log/juju/machine-x.log (where x is the machine number) on the controller around the times that the macaroon discharge fails?

15

@timClicks yes that is the process I used

@thumper I looked in /var/log/juju/machine-0.log initially, there is nothing in there related to this.

I can create error messages in that file by forging connections to controller:17070 with curl, whereby I cause some websocket errors. But attempting to log in using the juju client produces no message at all. This is where my confusion started. Is this a client error or the result of a server one? I have no evidence either way unfortunately. The latest messages in there right now appear to be from the last time I rebooted the controller;

root@bigmetal01:~# tail /var/log/juju/machine-0.log
2019-10-08 19:02:49 ERROR juju.worker.dependency engine.go:663 "api-caller" manifold worker returned unexpected error: [b5fb8e] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused
2019-10-08 19:02:50 WARNING juju.mongo open.go:160 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: connect: connection refused
2019-10-08 19:02:50 WARNING juju.mongo open.go:160 mongodb connection failed, will retry: dial tcp 192.168.200.233:37017: connect: connection refused
2019-10-08 19:03:25 INFO juju.cmd supercommand.go:57 running jujud [2.5.8 gc go1.11.11]
2019-10-08 19:03:25 DEBUG juju.cmd supercommand.go:58   args: []string{"/var/lib/juju/tools/machine-0/jujud", "machine", "--data-dir", "/var/lib/juju", "--machine-id", "0", "--debug", "--verbose"}
2019-10-08 19:03:25 DEBUG juju.utils gomaxprocs.go:24 setting GOMAXPROCS to 4
2019-10-08 19:03:25 DEBUG juju.agent agent.go:545 read agent config, format "2.0"
2019-10-08 19:03:25 INFO juju.cmd.jujud agent.go:133 setting logging config to "<root>=WARNING;unit=DEBUG"
2019-10-08 19:03:26 ERROR juju.worker.dependency engine.go:663 "api-caller" manifold worker returned unexpected error: [b5fb8e] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused
2019-10-08 19:03:30 ERROR juju.worker.dependency engine.go:663 "api-caller" manifold worker returned unexpected error: [b5fb8e] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused

Oh, you may notice I added a ā€˜ā€“verboseā€™ to jujud, it was a shot in the dark and apparently achieved nothing.

16

@thumper, @timClicks, are you able to suggest anything further please? I am really stuck unable to add compute nodes to this cluster and I really hope not to have to rebuild everything.

Many thanks in advance!

17

Not one to give upā€¦ @cmars @timClicks @thumper CFI

I have managed to replace the jujud on my controller with an unstripped version and I can attach the debugger (dlv). I have traced the authentication flow as far as this;

> github.com/juju/juju/apiserver/stateauthenticator.(*localLoginHandlers).serveLoginPost() ./locallogin.go:114 (PC: 0x2063782)
Warning: debugging optimized function
   109:         }
   110:
   111:         // Provide the client with a macaroon that they can use to
   112:         // prove that they have logged in, and obtain a discharge
   113:         // macaroon.
=> 114:         m, err := h.authCtxt.CreateLocalLoginMacaroon(userTag)
   115:         if err != nil {
   116:                 return nil, err
   117:         }
   118:         cookie, err := httpbakery.NewCookie(macaroon.Slice{m})
   119:         if err != nil {

At this point the client times out with a ā€œcontext deadline exceededā€ message. I will persist, but if there are any clues as to where I should be looking please post them :slight_smile:

1 Like
18

@cmars @timClicks @thumper @uros-jovanovic

Here is where I got to, although I donā€™t really understand what Iā€™m looking at yet.

> github.com/juju/juju/vendor/gopkg.in/macaroon-bakery.v2-unstable/bakery.checkNeedDeclared() /home/ubuntu/go/src/github.com/juju/juju/vendor/gopkg.in/macaroon-bakery.v2-unstable/bakery/service.go:375 (PC: 0x8efaf6)
Warning: debugging optimized function
   370:         if len(needDeclared) == 0 {
   371:                 return nil, fmt.Errorf("need-declared caveat with no required attributes")
   372:         }
   373:         cavInfo.Condition = arg[i+1:]
   374:         caveats, err := checker.CheckThirdPartyCaveat(cavInfo)
=> 375:         if err != nil {
   376:                 return nil, errgo.Mask(err, errgo.Any)
   377:         }
   378:         declared := make(map[string]bool)
   379:         for _, cav := range caveats {
   380:                 if cav.Location != "" {
(dlv) whatis cavInfo
*github.com/juju/juju/vendor/gopkg.in/macaroon-bakery.v2-unstable/bakery.ThirdPartyCaveatInfo
(dlv) whatis cavInfo.Condition
string
(dlv) print cavInfo.Condition
"is-authenticated-user admin"
(dlv) print err
error(*github.com/juju/juju/vendor/gopkg.in/macaroon-bakery.v2-unstable/httpbakery.Error) *{
    Code: "interaction required",
    Message: "interaction required",
    Info: *github.com/juju/juju/vendor/gopkg.in/macaroon-bakery.v2-unstable/httpbakery.ErrorInfo {
            Macaroon: *github.com/juju/juju/vendor/gopkg.in/macaroon%2ev2-unstable.Macaroon nil,
            MacaroonPath: "",
            CookieNameSuffix: "",
            VisitURL: "/auth/login?waitid=dbe4c0905d38656bc6daa20f",
            WaitURL: "/auth/wait?waitid=dbe4c0905d38656bc6daa20f",},
    version: version0 (0),}

I ā€œis-authenticated-userā€. Is this perhaps something upstream?

19

Ok, ignore that nonsense above. I understand the macaroons better now and can read the Golang :blush:

When I follow the login process on jujud, it runs through two iterations - once before and once after I am prompted for my password. On both iterations it ends up at;

github.com/juju/juju/vendor/gopkg.in/macaroon-bakery.v2-unstable/httpbakery/client.go:701

Apparently there is no macaroon presented on the second pass? This despite jujud apparently having authenticated me. My attention is now on what the client is doing.

20

@timClicks @thumper Iā€™ve managed to get roughly back to where I was before this problem appeared. To do this I reconstructed (as far as possible) all the missing files in .local/share/juju, specifically clouds.yaml, credentials.yaml and accounts.yaml as described by @thumper above. I previously set a new admin password in mongodb on the controller, so I put this password in accounts.yaml.

Now I am logged in apparently.

Iā€™m scared to touch it further, so I tried to create something reproduceable with a new controller. Hereā€™s what I did;

testuser@juju:~$ cat mycloud.yaml
clouds:
  mycloud:
    type: maas
    auth-types: [oauth1]
    endpoint: http://192.168.200.251:5240/MAAS
testuser@juju:~$ juju add-cloud mycloud mycloud.yaml
testuser@juju:~$ juju add-credential mycloud
Enter credential name: admin

Using auth-type "oauth1".

Enter maas-oauth:

Credential "admin" added locally for cloud "mycloud".

testuser@juju:~$ juju bootstrap mycloud mycontroller
Creating Juju controller "mycontroller" on mycloud
Looking for packaged Juju agent version 2.5.8 for amd64
Launching controller instance(s) on mycloud...
 - gq7cfn (arch=amd64 mem=4G cores=2)
Installing Juju agent on bootstrap instance
Fetching Juju GUI 2.15.0
Waiting for address
Attempting to connect to 192.168.200.32:22
Connected to 192.168.200.32
Running machine configuration script...
Bootstrap agent now started
Contacting Juju controller at 192.168.200.32 to verify accessibility...
Bootstrap complete, "mycontroller" controller now available
Controller machines are in the "controller" model
Initial model "default" added
testuser@juju:~$ juju change-user-password
new password:
type new password again:
ERROR recording macaroon: connecting to API: cannot get discharge from "https://192.168.200.32:17070/auth": failed to acquire macaroon after waiting: third party refused discharge: discharging macaroon: interaction required

Snookered :slight_smile:

testuser@juju:~$ juju destroy-controller mycontroller
WARNING! This command will destroy the "mycontroller" controller.
This includes all machines, applications, data and other resources.

Continue? (y/N):y
ERROR failed to destroy controller "mycontroller"

If the controller is unusable, then you may run

    juju kill-controller

to forcibly destroy the controller. Upon doing so, review
your cloud provider console for any resources that need
to be cleaned up.

ERROR cannot connect to API: invalid entity name or password (unauthorized access)

What did I do wrong?