This means that the cloud credential that you’ve used on the models is no longer valid from the perspective of your cloud provider. For example, it could have expired.
You need to make sure that your credential is valid from cloud provider perspective, for e.g. change password. What you need to do depends on what went wrong with your existing credential.
Then, you’d need to update that credential on your local client using ‘juju add-credential --replace’.
Once done, you need to let your controller know what the updated credential content is using ‘juju update-credential’.
Note that both local and remote credential need to have the same name for this update to succeed. If you do not know the name of the remote credential, first run ‘show-model’ for any of the suspend models. Credential name will be in the output.
This means that we cannot update controller credential because it is not valid for at least one model that uses it.
There are 2 things that you can do based on your requirements.
The credential is not valid for openstack-devzero model because when Juju uses that credential, it can not reach the cloud instances for the listed machines. This is most likely because the instances have been deleted via cloud provider interface? If this is the case and you definitely don’t want to see these machines again, you need to manually force remove these machines "remove-machine [0,1…] --force’. Make sure you are on the right model. Note that this operation may fail depending on what Juju version you are using. However, once the machines are removed, update credential on controller should succeed.
You may change the cloud credential that openstack-devzero uses if you have another valid credential. Use ‘set-credential’ command to achieve this. However, even if your new credential is valid, you may still have issues setting new cloud credential on the model if the cloud instances for these machines have been removed. In this case, you need to either ‘remove-machine --force’ or set credential for this model to “” using db surgery. Once this model does not use the credential you want to update, the update of credential on the controller for other models should succeed.
Sorry for the long post, but I hope someone finds this useful someday
I don’t know why this controller ended up being so unhappy.
In the state that it was in, any command I issued was simply being ignored even using –force and –no-wait
What I ended up doing was creating an additional credential for the maas cloud and then switching through each model and telling the controller to use the new credential
root@maas-region-ctl:~# juju add-credential maas.devzero.co.za
Enter credential name: new
Using auth-type "oauth1".
Enter maas-oauth:
Credential "new" added locally for cloud "maas.devzero.co.za".
root@maas-region-ctl:~# juju set-credential -m openstack-devzero maas.devzero.co.za new
Did not find credential remotely. Looking locally...
Uploading local credential to the controller.
Changed cloud credential on model "openstack-devzero" to "new".
root@maas-region-ctl:~#
root@maas-region-ctl:~# juju status
Model Controller Cloud/Region Version SLA Timestamp Notes
openstack-devzero juju.maas.devzero.co.za maas.devzero.co.za 2.4.6 unsupported 12:29:17Z attempt 5 to destroy model failed (will retry): model not empty, found 2 machines (model not empty)
Machine State DNS Inst id Series AZ Message
3 stopped 165.117.214.184 efcxrt bionic default Deployed
Finally model was destroyed. However all the other models were still suspended, so I cycled through each of them and added the “new” credential
root@maas-region-ctl:~# juju models
Controller: juju.maas.devzero.co.za
Model Cloud/Region Type Status Machines Cores Units Access Last connection
awx-ansible maas.devzero.co.za maas suspended 4 8 4 admin 9 hours ago
controller maas.devzero.co.za maas suspended 1 4 - admin just now
default maas.devzero.co.za maas suspended 0 - - admin 2019-05-09
devzero-openstack maas.devzero.co.za maas suspended 18 - 24 admin 9 hours ago
elastic-test maas.devzero.co.za maas suspended 5 14 5 admin 9 hours ago
ha-awx-ansible maas.devzero.co.za maas suspended 4 16 4 admin 18 hours ago
kuber-core maas.devzero.co.za maas suspended 4 14 8 admin 8 hours ago
root@maas-region-ctl:~# juju set-credential -m awx-ansible maas.devzero.co.za new
Found credential remotely, on the controller. Not looking locally...
Changed cloud credential on model "awx-ansible" to "new".
root@maas-region-ctl:~#
===snip ===
root@maas-region-ctl:~# juju models
Controller: juju.maas.devzero.co.za
Model Cloud/Region Type Status Machines Cores Units Access Last connection
awx-ansible maas.devzero.co.za maas available 4 8 4 admin 9 hours ago
controller maas.devzero.co.za maas available 1 4 - admin just now
default maas.devzero.co.za maas available 0 - - admin 2019-05-09
elastic-test maas.devzero.co.za maas available 5 14 5 admin 9 hours ago
ha-awx-ansible maas.devzero.co.za maas available 4 16 4 admin 18 hours ago
kuber-core maas.devzero.co.za maas available 4 14 8 admin 8 hours ago
Indeed, once you have set-credential on the ‘faulty’ model to the new one, you could have re-run ‘update-credential’ command for the old credential and that operation would have succeeded. It would have saved you from ‘set-credential’ individually on all other models.
juju update-credential myvscloud bidwell
This operation can be applied to both a copy on this client and to the one on a controller.
Do you want to update credential "bidwell" on cloud "myvscloud" on:
1. client only (--client)
2. controller "myvscontroller" only (--controller myvscontroller)
3. both (--client --controller myvscontroller)
Enter your choice, or type Q|q to quit: 2
Credential valid for:
mariadbs
meu
wordpress
controller
ssvpns
Credential invalid for:
default:
couldn't find instance "juju-b6e4b2-7" for machine 7
WARNING Controller credential "bidwell" for user "admin" for cloud "myvscloud" on controller "myvscontroller" not updated: some models are no longer visible.
Failed models may require a different credential.
Use ‘juju set-credential’ to change credential for these models before repeating this update.
I don’t know why this controller ended up being so unhappy.