Juju destroy-model takes too long (sometimes returns timeout)

juju
,
1

Hi folks,

While developing K8s charms I create and destroy models regularly, but sometimes juju takes a very long time to perform the action or even fails with timeout.

$ juju destroy-model -y mysql --force --no-wait --destroy-storage && juju switch controller
Destroying model
Waiting for model to be removed, 1 application(s), 3 volume(s), 3 filesystems(s).....
Waiting for model to be removed, 1 application(s)...
Waiting for model to be removed.................................................
................................................................................
................................................................................
................................................................................
......................................................................

While trying to remove the model a watch on juju status returns:

Cada 1,0s: juju status --color                                                                                                                                                                              miguelito: Thu Mar 25 15:45:34 2021

Model  Controller  Cloud/Region        Version  SLA          Timestamp       Notes
mysql  mk8s        microk8s/localhost  2.9-rc7  unsupported  15:45:35-03:00  attempt 6 to destroy model failed (will retry):  model not empty, found 1 volume, 1 filesystem (model not empty)

Model "admin/mysql" is empty.

I would like to understand why this happens. Which would be the best way to troubleshoot this?
I am using:

$  juju version
2.9-rc7-focal-amd64

thanks in advance!

2

Juju asks k8s to delete the namespace hosting the model and also delete any persistent volume claims. Juju will wait until this completes.
I have seen many times that k8s takes forever to remove these artifacts. You can use kubectl to confirm that the namespace is “terminating” etc to see that the wait is for k8s to complete the cleanup.

1 Like
3

My experience is that if you remove the machines first with
juju remove-machine X --force

Then this process is much less error prone.

4

Thanks @wallyworld

I forgot to mention that I am using microk8s version:

$ snap info microk8s  | grep installed
installed:          v1.20.4                    (2074) 218MB classic
5

Hi folks,

I encountered this issue again and discovered that the solution involves removing the model and its references in the database. To accomplish this, log into the Mongo database and execute the script below with your model ID (which can be found in .local/share/juju/models.yaml):

var db = db.getSiblingDB('juju');
var cleanup_model_uuids = [
    "your-model-id-here"
];

db.getCollectionNames().forEach(function(coll) {
    var count = db.getCollection(coll).count({"model-uuid": {$in: cleanup_model_uuids}});
    if (count > 0) {
        var deleted = db.getCollection(coll).remove({"model-uuid": {$in: cleanup_model_uuids}});
        print("Deleted " + deleted + " objects from " + coll);
    }
});

Once that’s done, you’ll need to manually remove all resources in the provider (VMs, containers, etc.) and also delete the associated entries in .local/share/juju/models.yaml.

2 Likes