We have an issue with the postgresql charm on lxd (Channel: 14/stable Rev: 429). The charm has entered en error state and the debug-log shows that the leadership-tracker fails.
This also prevents us from running any actions on the unit, so we can’t effectively get the password to get into the database to perform manual administration of the unit.
I have increased the model logging to debug and this is what I get from the model.
==> /var/log/juju/unit-postgresql-1.log <==
2024-12-21 10:56:51 DEBUG juju.worker.uniter.relation statetracker.go:221 unit "postgresql/1" (leader=false) entered scope for relation "postgresql:s3-parameters s3-integrator:s3-credentials"
2024-12-21 10:56:51 DEBUG juju.worker.dependency engine.go:618 "leadership-tracker" manifold worker stopped: leadership failure: lease operation timed out
stack trace:
lease operation timed out
github.com/juju/juju/worker/leadership.(*Tracker).refresh:187: leadership failure
github.com/juju/juju/worker/leadership.(*Tracker).isLeader:259:
github.com/juju/juju/worker/leadership.(*Tracker).resolveClaim:274:
github.com/juju/juju/worker/leadership.(*Tracker).loop:158:
2024-12-21 10:56:51 ERROR juju.worker.dependency engine.go:695 "leadership-tracker" manifold worker returned unexpected error: leadership failure: lease operation timed out
2024-12-21 10:56:51 INFO juju.worker.uniter uniter.go:347 unit "postgresql/1" shutting down: catacomb 0xc000b74900 is dying
2024-12-21 10:56:51 DEBUG juju.worker.dependency engine.go:603 "uniter" manifold worker completed successfully
2024-12-21 10:56:54 DEBUG juju.worker.dependency engine.go:580 "leadership-tracker" manifold worker started at 2024-12-21 10:56:54.321980328 +0000 UTC
==> /var/log/juju/machine-15.log <==
2024-12-21 10:56:54 DEBUG juju.worker.leadership tracker.go:125 postgresql/1 making initial claim for postgresql leadership
==> /var/log/juju/unit-postgresql-1.log <==
2024-12-21 10:56:54 DEBUG juju.worker.dependency engine.go:580 "uniter" manifold worker started at 2024-12-21 10:56:54.332854818 +0000 UTC
==> /var/log/juju/machine-15.log <==
2024-12-21 10:56:54 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-postgresql-1
2024-12-21 10:56:54 INFO juju.agent.tools symlinks.go:40 was a symlink, now looking at /var/lib/juju/tools/3.3.6-ubuntu-amd64
2024-12-21 10:56:54 DEBUG juju.agent.tools symlinks.go:50 target tools path /var/lib/juju/tools/3.3.6-ubuntu-amd64/jujuc
2024-12-21 10:57:24 INFO juju.worker.leadership tracker.go:194 postgresql/1 promoted to leadership of postgresql
==> /var/log/juju/unit-postgresql-1.log <==
2024-12-21 10:57:24 DEBUG juju.worker.uniter.relation statetracker.go:221 unit "postgresql/1" (leader=true) entered scope for relation "postgresql:upgrade"
2024-12-21 11:01:46 DEBUG juju.worker.uniter.relation statetracker.go:221 unit "postgresql/1" (leader=false) entered scope for relation "postgresql:database-peers"
2024-12-21 11:01:46 DEBUG juju.worker.dependency engine.go:618 "leadership-tracker" manifold worker stopped: leadership failure: lease operation timed out
stack trace:
lease operation timed out
github.com/juju/juju/worker/leadership.(*Tracker).refresh:187: leadership failure
github.com/juju/juju/worker/leadership.(*Tracker).loop:153:
2024-12-21 11:01:46 ERROR juju.worker.dependency engine.go:695 "leadership-tracker" manifold worker returned unexpected error: leadership failure: lease operation timed out
2024-12-21 11:01:46 INFO juju.worker.uniter uniter.go:347 unit "postgresql/1" shutting down: catacomb 0xc000b74d80 is dying
2024-12-21 11:01:46 DEBUG juju.worker.dependency engine.go:603 "uniter" manifold worker completed successfully
2024-12-21 11:01:49 DEBUG juju.worker.dependency engine.go:580 "leadership-tracker" manifold worker started at 2024-12-21 11:01:49.721444944 +0000 UTC
==> /var/log/juju/machine-15.log <==
2024-12-21 11:01:49 DEBUG juju.worker.leadership tracker.go:125 postgresql/1 making initial claim for postgresql leadership
==> /var/log/juju/unit-postgresql-1.log <==
2024-12-21 11:01:49 DEBUG juju.worker.dependency engine.go:580 "uniter" manifold worker started at 2024-12-21 11:01:49.73206939 +0000 UTC
==> /var/log/juju/machine-15.log <==
2024-12-21 11:01:49 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-postgresql-1
2024-12-21 11:01:49 INFO juju.agent.tools symlinks.go:40 was a symlink, now looking at /var/lib/juju/tools/3.3.6-ubuntu-amd64
2024-12-21 11:01:49 DEBUG juju.agent.tools symlinks.go:50 target tools path /var/lib/juju/tools/3.3.6-ubuntu-amd64/jujuc
A note or observation relevant to the context is that we are using the s3 integrator to backup to s3. Perhaps if something goes wrong there could lead to this situation?
Any help to get the unit out of the error state much welcomed.