I’m haveing major issues with updating credentials in our vsphere cloud/controller.
$ juju update-credentials my-vsphere mycreds
This operation can be applied to both a copy on this client and to the one on a controller.
Do you want to update credential "mycreds" on cloud "my-vsphere" on:
1. client only (--client)
2. controller "jimm.example.com" only (--controller jimm.example.com)
3. both (--client --controller jimm.example.com)
Enter your choice, or type Q|q to quit: 2
Credential valid for:
kjelle
ats-dev
nextcloud-1
gitlabrunners
atsrti
conan-config-repo
basehost2
newmodel2
centos7test
mzhxtqbuildenv
Credential invalid for:
basehosts:
couldn't find instance "juju-2af9f8-12" for machine 12
couldn't find instance "juju-2af9f8-13" for machine 13
couldn't find instance "juju-2af9f8-16" for machine 16
couldn't find instance "juju-2af9f8-0" for machine 0
couldn't find instance "juju-2af9f8-9" for machine 9
dcvtest:
couldn't find instance "juju-d7991d-0" for machine 0
couldn't find instance "juju-d7991d-1" for machine 1
jujuadmin:
couldn't find instance "juju-ff42ff-0" for machine 0
logging-hpc:
couldn't find instance "juju-2bdf83-0" for machine 0
mellanox-yumrepo:
couldn't find instance "juju-3cbc6c-0" for machine 0
newmodel1:
couldn't find instance "juju-dcd742-2" for machine 2
Failed models may require a different credential.
Use ‘juju set-credential’ to change credential for these models before repeating this update.
I can’t see any problems with my credentials (which as you can see works for other models)
The models are all running OK and instances are OK. I’ve rebooted one instance to see if the error would go away, it doesn’t.
I’ve tried running set-credential, but that doesn’t work… It looks like this then:
$ juju set-credential my-vsphere mycreds
Found credential remotely, on the controller. Not looking locally…
ERROR could not set model credential: model 5d2b18e8-18d6-4e78-83ca-c9f0692af9f8 already uses credential my-vsphere/ERIK@example.com/mycreds
I’m totally lost here and I’ve never seen this behaviour before…
@erik-lonroth shot in the dark… are you able to see machine details from juju status? If so, note those IP addrs. You said “I’ve rebooted one instance” implying you can get to an instance. If that’s true, does the IP of that instance match what juju status thinks that machine IP is?
I ask because I’ve had trouble in the past when my vsphere controller changed IP. In my case, i changed the relevant api-endpoint in ~/.local/share/juju/controllers.yaml to the right controller ip and things like update-creds worked for me again.
It seems that the models that are affected lives in a specific vsphere datacenter (juju: Region). Can this somehow be related to the credential not understanding how to update that if there are more than one region?
When a credential is updated to change the password or api secret or whatever, juju runs various checks to ensure the credential is valid. One of the checks runs through each machine in the model to check that a corresponding cloud instance exists. It seems that various models have orphaned machines in the model, and this is causing the credential check to fail.
Side note - I think the check is too strict since if one or two machines are orphaned and the rest are ok, the credential itself is probably still valid.
So what the command is recommending to fix the problem is to use set-credential to add an entirely different credential to the affected models. The reasoning is that if a credential works for some models and not others, those other models would need a different credential.
The error you are seeing is that you are trying to apply the same credential to a model using set-credential instead of setting a new credential.
Given the issues highlighted by the credential validation, one important step is to look at the orphaned machines and clean that up - if the cloud instance really has gone away, you will need to remove the machine from the juju model. And given the way the checks are currently done, this does need to be done before update-credential can be used on those highlighted models.
As stated above, I think this issue highlights the need to look at how the credential check is done against each model which uses it and not let issues unrelated to the credential validity affect the update.
When checking the validity of a credential, juju queries the underlying cloud to gather all running instances. It compares this set with what is recorded in the juju model. Errors like
couldn't find instance "juju-2af9f8-12" for machine 12
means that juju has a record in the model for machine 12 with an instance id of juju-2af9f8-12 in the cloud, but that instance id cannot be found looking at the running machines in the cloud.
All unit may work, but these may well be deployed to machines other than the ones listed as being orphaned. What does machine 12 show in juju status --format yaml? Is it running and alive? Or maybe it is dying and the cloud instance has already been removed but there’s something has has prevented the juju model being cleaned up. Or, I wonder what running lxc list shows… is juju-2af9f8-12 shown as running? If so, juju is somehow not seeing that instance when it is doing the credential check.
I got this from the support at Canonical, which resolved my issue
Hi Erik,
Under some circumstances, juju update-credential won’t work, and it’s instead necessary to use “juju add-credential” to add a completely new credential, and then set the default for the cloud with “juju default-credential”.
Please could you try that and advise if you’re still stuck?
Regards,
Performing this resolved the issue.
I think this is critical information for managing credentials with vsphere @tmihoc
Thanks, @erik-lonroth, I’ve updated the vSphere doc to reflect this. PS For now it’s just a note in the Add a credential section. I’m planning to find it a better place soon.
