I have a situation where a few LXC machine agents, following a maintenance, have got into some issues reconnecting with the controller (2.9.38).
When I’m debugging this - this is what I see in the log from /var/log/jujumachine-0.log
Pay attention to the authentication error: Failed to connect to controller: invalid entity name or password (unauthorized access)
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:580 "api-config-watcher" manifold worker started at 2023-03-09 08:43:09.85013368 +0000 UTC
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:580 "upgrade-check-gate" manifold worker started at 2023-03-09 08:43:09.850202374 +0000 UTC
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:580 "upgrade-check-flag" manifold worker started at 2023-03-09 08:43:09.850394172 +0000 UTC
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:580 "upgrade-steps-gate" manifold worker started at 2023-03-09 08:43:09.850785544 +0000 UTC
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:580 "clock" manifold worker started at 2023-03-09 08:43:09.851598108 +0000 UTC
2023-03-09 08:43:09 DEBUG juju.worker.apicaller connect.go:129 connecting with old password
2023-03-09 08:43:09 DEBUG juju.worker.introspection worker.go:135 introspection worker listening on "@jujud-machine-0"
2023-03-09 08:43:09 DEBUG juju.worker.introspection worker.go:161 stats worker now serving
2023-03-09 08:43:09 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://192.168.151.64:17070/model/c85701ca-70ff-405c-8b94-b7876019051b/api"
2023-03-09 08:43:09 INFO juju.api apiclient.go:687 connection established to "wss://192.168.151.64:17070/model/c85701ca-70ff-405c-8b94-b7876019051b/api"
2023-03-09 08:43:09 DEBUG juju.worker.apicaller connect.go:160 [c85701] failed to connect
2023-03-09 08:43:09 ERROR juju.worker.apicaller connect.go:209 Failed to connect to controller: invalid entity name or password (unauthorized access)
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:618 "api-caller" manifold worker stopped: [c85701] "machine-0" cannot open api: connection permanently impossible
stack trace:
github.com/juju/juju/worker/apicaller.init:42: connection permanently impossible
github.com/juju/juju/worker/apicaller.ManifoldConfig.startFunc.func1:97: [c85701] "machine-0" cannot open api
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:603 "upgrade-check-flag" manifold worker completed successfully
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:603 "is-not-controller-flag" manifold worker completed successfully
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:603 "upgrade-check-gate" manifold worker completed successfully
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:603 "agent" manifold worker completed successfully
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:603 "is-controller-flag" manifold worker completed successfully
2023-03-09 08:43:09 INFO juju.worker.stateconfigwatcher manifold.go:120 tomb dying
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:603 "upgrade-steps-gate" manifold worker completed successfully
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:603 "termination-signal-handler" manifold worker completed successfully
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:603 "clock" manifold worker completed successfully
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:603 "api-config-watcher" manifold worker completed successfully
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:603 "syslog" manifold worker completed successfully
2023-03-09 08:43:09 DEBUG juju.worker.dependency engine.go:603 "state-config-watcher" manifold worker completed successfully
2023-03-09 08:43:09 INFO cmd supercommand.go:544 command finished
2023-03-09 08:43:09 DEBUG juju.cmd.jujud main.go:283 jujud complete, code 0, err <nil>
Restarting the juju machine-agent (systemctl restart jujud-machine-0.service) does not help.
When comparing the “/var/lib/juju/agents/machine-0/agent.conf” with working another working agent, I see that some information in the broken one is missing the apipassword key as shown below:
Broken:
Working:
How can I get my agent to re-connect with my controller?