Juju resources and the upgrade-charm event have become a point of contention for us as the number of units in our models have increased.
upgrade-charm hook event is dumb in the sense that it is not contextually aware of what is being upgraded. There are a few issues that stem from this that make things really clunky when trying to mitigate resources from a charm author perspective.
In the case of a charm with one or more resources, the charm has no way of knowing which resource is being upgraded, or if it is the charm code itself that is being upgraded.
Imagine a case where a charm wants to preform some operation on the resource only if it has changed, and not blindly when the upgrade event/hook fires (possibly only the charm code is being upgraded and we don’t want to do anything to the resources of the charm).
Per this conversation on upgrading charm vs resources it seems there is a path forward where users are storing and checking the sha of the resource to be able to determine if it has changed.
This makes good sense logically, with the exception that
resource-get has to run for each resource, for each unit, in order to get the resource local to the unit to be able to check and compare the sha, possibly needlessly if only the charm code has changed.
Using this method, considering a model of 50 units, the juju controllers get entirely stomped handing out a resource to a bunch of asking units. The next thing that happens after the controllers basically go down is that the environment gets stuck in an unusable and unstable state for quite some time while units wait for their resource from the controllers and for the controllers to resume/recover from the load.
With anywhere from 50-200 primary charm units per application in each of our models, this hiccup has been biting us badly, causing us to entirely re-provision clusters in order to upgrade them – we can’t use the the upgrade hook combined with charm resources because takes everything down.
Ideas for a solution
Provide a way to check the sha of a resource without pulling it down to the unit. I think this could be accomplished via:
- Add context to the upgrade-charm hook that would expose the sha of each resource
- Create a
resource_info()function that could return the info (sha) of the desired resource(s)
- Create resource centric hook events that fire when a resource has changed