Config-changed no longer running after reboot

Hi,

Recently, I updated the Juju version in our production environment from 2.7.2 -> 2.7.3. Afterwards, I noticed that config-changed is no longer being called after rebooting a machine. The machine reboots fine, and the jujud service starts, the machine isn’t in error state,but config-changed is never ran.

Is this a change/bug in 2.7.3? Do I need to upgrade again?

(filtered, removed the connection messages) logs on the machine after reboot for reference:

2020-06-16 21:42:36 INFO juju.cmd supercommand.go:83 running jujud [2.7.3 9da5b593fa1d096220230143f1e1fea5c4eb56ad gc go1.12.17]
2020-06-16 21:42:36 DEBUG juju.cmd supercommand.go:84   args: []string{"/var/lib/juju/tools/unit-x86-cnr-0/jujud", "unit", "--data-dir", "/var/lib/juju", "--unit-name", "x86-cnr/0", "--debug"}
2020-06-16 21:42:36 DEBUG juju.agent agent.go:545 read agent config, format "2.0"
2020-06-16 21:42:36 INFO juju.cmd.jujud agent.go:133 setting logging config to "<root>=INFO;unit=DEBUG"
2020-06-16 21:42:36 INFO juju.worker.upgradesteps worker.go:72 upgrade steps for 2.7.3 have already been run.
2020-06-16 21:42:36 INFO juju.worker.upgrader upgrader.go:155 abort check blocked until version event received
2020-06-16 21:42:36 INFO juju.worker.upgrader upgrader.go:161 unblocking abort check
2020-06-16 21:42:36 INFO juju.worker.upgrader upgrader.go:194 desired agent binary version: 2.7.3
2020-06-16 21:42:36 INFO juju.worker.migrationminion worker.go:139 migration phase is now: NONE
2020-06-16 21:42:36 INFO juju.worker.logger logger.go:118 logger worker started
2020-06-16 21:42:36 INFO juju.worker.leadership tracker.go:194 x86-cnr/0 promoted to leadership of x86-cnr
2020-06-16 21:42:36 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-x86-cnr-0
2020-06-16 21:42:36 INFO juju.agent.tools symlinks.go:40 was a symlink, now looking at /var/lib/juju/tools/2.7.3-focal-amd64
2020-06-16 21:42:36 INFO juju.worker.uniter uniter.go:246 unit "x86-cnr/0" started
2020-06-16 21:42:36 INFO juju.worker.uniter uniter.go:285 hooks are retried false
2020-06-16 21:42:37 INFO juju.worker.upgrader upgrader.go:194 desired agent binary version: 2.7.3

After which the machine just sits in update_status polling.

This change was actually made in Juju 2.5.0, although I did have to dig to see when the change occurred. It is in the 2.5.0 Release Notes.

This was changed to stop the stampeding herd problem that would occur when the controllers were restarted.

In Juju 2.8.0 another change was introduced to support the sort of use case you are talking about. There are a number of charms in OpenStack that need coordination when the machine reboots. From 2.8.0 and on, when a machine with a unit is restarted, the start hook for the unit is called. This only happens the first time the unit agent is started for that agent.

Thanks for the dig up, thumper. We are able to make do with the start hook