Ha! Yeah that got trashed already I only have controllers.yaml here now.
I followed Bui Ha’s instruction to reset the admin password in the database, which appears to have worked in that I get the macaroon error when I log in as admin with the new password I set;
routergod@juju:~/.local/share/juju$ juju login -u admin
please enter password for admin on openstack:
ERROR cannot log into controller "openstack": cannot get discharge from "https://192.168.200.233:17070/auth": failed to acquire macaroon after waiting: third party refused discharge: discharging macaroon: interaction required
@routergod I went looking for the bug as @cmars asked for debug output. Something is clearly not working here, and I’d love to know what.
I take it you have ssh access to the controller machine since you were able to reset the admin password. Is that right?
If so, is there anything in /var/log/juju/machine-x.log (where x is the machine number) on the controller around the times that the macaroon discharge fails?
@thumper I looked in /var/log/juju/machine-0.log initially, there is nothing in there related to this.
I can create error messages in that file by forging connections to controller:17070 with curl, whereby I cause some websocket errors. But attempting to log in using the juju client produces no message at all. This is where my confusion started. Is this a client error or the result of a server one? I have no evidence either way unfortunately. The latest messages in there right now appear to be from the last time I rebooted the controller;
root@bigmetal01:~# tail /var/log/juju/machine-0.log
2019-10-08 19:02:49 ERROR juju.worker.dependency engine.go:663 "api-caller" manifold worker returned unexpected error: [b5fb8e] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused
2019-10-08 19:02:50 WARNING juju.mongo open.go:160 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: connect: connection refused
2019-10-08 19:02:50 WARNING juju.mongo open.go:160 mongodb connection failed, will retry: dial tcp 192.168.200.233:37017: connect: connection refused
2019-10-08 19:03:25 INFO juju.cmd supercommand.go:57 running jujud [2.5.8 gc go1.11.11]
2019-10-08 19:03:25 DEBUG juju.cmd supercommand.go:58 args: []string{"/var/lib/juju/tools/machine-0/jujud", "machine", "--data-dir", "/var/lib/juju", "--machine-id", "0", "--debug", "--verbose"}
2019-10-08 19:03:25 DEBUG juju.utils gomaxprocs.go:24 setting GOMAXPROCS to 4
2019-10-08 19:03:25 DEBUG juju.agent agent.go:545 read agent config, format "2.0"
2019-10-08 19:03:25 INFO juju.cmd.jujud agent.go:133 setting logging config to "<root>=WARNING;unit=DEBUG"
2019-10-08 19:03:26 ERROR juju.worker.dependency engine.go:663 "api-caller" manifold worker returned unexpected error: [b5fb8e] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused
2019-10-08 19:03:30 ERROR juju.worker.dependency engine.go:663 "api-caller" manifold worker returned unexpected error: [b5fb8e] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused
Oh, you may notice I added a ‘–verbose’ to jujud, it was a shot in the dark and apparently achieved nothing.
@thumper, @timClicks, are you able to suggest anything further please? I am really stuck unable to add compute nodes to this cluster and I really hope not to have to rebuild everything.
I have managed to replace the jujud on my controller with an unstripped version and I can attach the debugger (dlv). I have traced the authentication flow as far as this;
> github.com/juju/juju/apiserver/stateauthenticator.(*localLoginHandlers).serveLoginPost() ./locallogin.go:114 (PC: 0x2063782)
Warning: debugging optimized function
109: }
110:
111: // Provide the client with a macaroon that they can use to
112: // prove that they have logged in, and obtain a discharge
113: // macaroon.
=> 114: m, err := h.authCtxt.CreateLocalLoginMacaroon(userTag)
115: if err != nil {
116: return nil, err
117: }
118: cookie, err := httpbakery.NewCookie(macaroon.Slice{m})
119: if err != nil {
At this point the client times out with a “context deadline exceeded” message. I will persist, but if there are any clues as to where I should be looking please post them
Ok, ignore that nonsense above. I understand the macaroons better now and can read the Golang
When I follow the login process on jujud, it runs through two iterations - once before and once after I am prompted for my password. On both iterations it ends up at;
Apparently there is no macaroon presented on the second pass? This despite jujud apparently having authenticated me. My attention is now on what the client is doing.
@timClicks@thumper I’ve managed to get roughly back to where I was before this problem appeared. To do this I reconstructed (as far as possible) all the missing files in .local/share/juju, specifically clouds.yaml, credentials.yaml and accounts.yaml as described by @thumper above. I previously set a new admin password in mongodb on the controller, so I put this password in accounts.yaml.
Now I am logged in apparently.
I’m scared to touch it further, so I tried to create something reproduceable with a new controller. Here’s what I did;
testuser@juju:~$ cat mycloud.yaml
clouds:
mycloud:
type: maas
auth-types: [oauth1]
endpoint: http://192.168.200.251:5240/MAAS
testuser@juju:~$ juju add-cloud mycloud mycloud.yaml
testuser@juju:~$ juju add-credential mycloud
Enter credential name: admin
Using auth-type "oauth1".
Enter maas-oauth:
Credential "admin" added locally for cloud "mycloud".
testuser@juju:~$ juju bootstrap mycloud mycontroller
Creating Juju controller "mycontroller" on mycloud
Looking for packaged Juju agent version 2.5.8 for amd64
Launching controller instance(s) on mycloud...
- gq7cfn (arch=amd64 mem=4G cores=2)
Installing Juju agent on bootstrap instance
Fetching Juju GUI 2.15.0
Waiting for address
Attempting to connect to 192.168.200.32:22
Connected to 192.168.200.32
Running machine configuration script...
Bootstrap agent now started
Contacting Juju controller at 192.168.200.32 to verify accessibility...
Bootstrap complete, "mycontroller" controller now available
Controller machines are in the "controller" model
Initial model "default" added
testuser@juju:~$ juju change-user-password
new password:
type new password again:
ERROR recording macaroon: connecting to API: cannot get discharge from "https://192.168.200.32:17070/auth": failed to acquire macaroon after waiting: third party refused discharge: discharging macaroon: interaction required
Snookered
testuser@juju:~$ juju destroy-controller mycontroller
WARNING! This command will destroy the "mycontroller" controller.
This includes all machines, applications, data and other resources.
Continue? (y/N):y
ERROR failed to destroy controller "mycontroller"
If the controller is unusable, then you may run
juju kill-controller
to forcibly destroy the controller. Upon doing so, review
your cloud provider console for any resources that need
to be cleaned up.
ERROR cannot connect to API: invalid entity name or password (unauthorized access)
I’m slightly in awe of your persistence. This isn’t a subsystem that I have a lot of familiarity with, but it certainly looks like you’ve encountered a significant bug.
I replied to the bug and noted this looks like 2.5. I’ve repeated the steps in 2.6 and 2.7 with it working correctly. I’m curious if we can test using the later client/controller combo? I know we did a bunch of solid work in 2.6 for the login/logout/auth processing.
After the latest notes in that bug I’m wondering if it’s something about how MAAS is setup that’s causing us to get different behavior? Can you speak to the MAAS setup? Are you using RBAC or somethings else non-default in the auth side we should be double checking?
@rick_h I have been trying to understand where in the apiserver code the absent macaroon would be issued. I can only find two candidates;
./apiserver/registration.go:61: m, err := h.ctxt.srv.authenticator.CreateLocalLoginMacaroon(userTag)
./apiserver/stateauthenticator/locallogin.go:114: m, err := h.authCtxt.CreateLocalLoginMacaroon(userTag)
That is, assuming CreateLocalLoginMacaroon() is relevant? I figure the first of these is in relation to the ‘juju register’ command.
The code in locallogin.go looks promising, but this authentication path is apparently not exercised when I use ‘juju change-user-password’. Rather, the authentication lands me here;
(dlv) bt
0 0x0000000002ec7a2f in github.com/juju/juju/apiserver/stateauthenticator.(*modelUserEntity).PasswordValid
at /home/ubuntu/go/src/github.com/juju/juju/apiserver/stateauthenticator/modeluser.go:119
1 0x0000000002928427 in github.com/juju/juju/apiserver/authentication.(*AgentAuthenticator).Authenticate
at /home/ubuntu/go/src/github.com/juju/juju/apiserver/authentication/agent.go:39
2 0x0000000002929de5 in github.com/juju/juju/apiserver/authentication.(*UserAuthenticator).Authenticate
at /home/ubuntu/go/src/github.com/juju/juju/apiserver/authentication/user.go:77
3 0x0000000002ec299a in github.com/juju/juju/apiserver/stateauthenticator.authenticator.Authenticate
at /home/ubuntu/go/src/github.com/juju/juju/apiserver/stateauthenticator/context.go:147
4 0x0000000002ec9364 in github.com/juju/juju/apiserver/stateauthenticator.(*authenticator).Authenticate
at <autogenerated>:1
5 0x0000000002ec0c8f in github.com/juju/juju/apiserver/stateauthenticator.(*Authenticator).checkCreds
at /home/ubuntu/go/src/github.com/juju/juju/apiserver/stateauthenticator/auth.go:152
6 0x0000000002ec05be in github.com/juju/juju/apiserver/stateauthenticator.(*Authenticator).AuthenticateLoginRequest
at /home/ubuntu/go/src/github.com/juju/juju/apiserver/stateauthenticator/auth.go:114
7 0x0000000002e17878 in github.com/juju/juju/apiserver.(*admin).authenticate
at /home/ubuntu/go/src/github.com/juju/juju/apiserver/admin.go:272
8 0x0000000002e157e9 in github.com/juju/juju/apiserver.(*admin).login
at /home/ubuntu/go/src/github.com/juju/juju/apiserver/admin.go:90
9 0x0000000002e15483 in github.com/juju/juju/apiserver.(*admin).Login
at /home/ubuntu/go/src/github.com/juju/juju/apiserver/admin.go:66
10 0x0000000000465015 in runtime.call512
at /snap/go/4286/src/runtime/asm_amd64.s:526
11 0x00000000004f0914 in reflect.Value.call
at /snap/go/4286/src/reflect/value.go:447
12 0x00000000004efbea in reflect.Value.Call
at /snap/go/4286/src/reflect/value.go:308
13 0x0000000000c9c70e in github.com/juju/juju/vendor/github.com/juju/rpcreflect.newMethod.func8
at /home/ubuntu/go/src/github.com/juju/juju/vendor/github.com/juju/rpcreflect/type.go:344
14 0x0000000000c9b652 in github.com/juju/juju/vendor/github.com/juju/rpcreflect.methodCaller.Call
at /home/ubuntu/go/src/github.com/juju/juju/vendor/github.com/juju/rpcreflect/value.go:129
15 0x0000000000c9d1f2 in github.com/juju/juju/vendor/github.com/juju/rpcreflect.(*methodCaller).Call
at <autogenerated>:1
16 0x0000000000ca19a8 in github.com/juju/juju/rpc.(*Conn).runRequest
at /home/ubuntu/go/src/github.com/juju/juju/rpc/server.go:571
17 0x0000000000466ab1 in runtime.goexit
at /snap/go/4286/src/runtime/asm_amd64.s:1333
None of AuthenticateLoginRequest() or anything called appears to do anything with macaroons.
What am I missing here? At what point should a macaroon be issued when I do;
Oh wow, ok. This explains a lot. I would expect most servers to run ntpd out of the box. There is an ntp charm you could try out maybe? For something baked into MAAS would have to hit them up. Might be worth checking their discourse.
I only have controllers.yaml here now.
