Firstly I’d like to echo many of the others and say “Thank you” for taking the time to write up your experience. I’d like to comment on a few points, but that doesn’t take away from all the rest.
As I’m sure you have noticed, Juju will automatically retry hook errors. However there is a command that you can use to tell the Juju controller (and model) that you have handled the situation.
juju resolved --no-retry <unit-name>
This will effectly remove the error state from the unit for the current hook execution. Juju then moves on to the next thing it would run for that unit. Frequently this is another hook. However, sometimes hooks are written in such a way that the next hook also enters an error state. You can continue to run the resolved command to move the agent along from each hook, effectively telling it that you have handled it. Once the queue of hooks have been executed, and resolved, the unit should enter its idle state. This would allow you to remove the application or upgrade the charm.
You may find that during the removal of the application, you’d also hit errors. You can mark these resolved too, and you should get to the stage where the application is removed without breaking the other applications it is related to.
This is definitely something I think we could do better about. Perhaps being able to set a policy for the model, so when you remove the last unit for a machine, it could leave the machine there. This should be very easy to implement.
As an operator of that type of model, you could still call remove-machine explicitly to get rid of the machine.
I like this idea, and is something definitely worth exploring.
There are other elements as well that I’m thinking on, but I wanted to explicitly comment on these.