[Azure] suspended since cloud credential is not valid

juju
#1

Hi there,

I’ve got a model on Azure with custom permissions (using the Service Provider method with juju). When creating models, or deploying bundles juju is able to do it’s job fine, but after a little while the models will go into the “suspended since cloud credential is not valid”. After that, any subsqeuent models are then also unable to do anything. See below:

Model          Controller        Cloud/Region      Version  SLA          Timestamp  Notes
custom-rg-net  azure-controller  azure/westeurope  2.9-rc5  unsupported  10:54:37Z  suspended since cloud credential is not valid

Model "admin/custom-rg-net" is empty.

I have a suspicion that there could be a missing permission on my service pricipal (I have added the permissions from the docs). I just need some guidance on a few points validating/invalidating this:

  • Is there somewhere where the invalidation reason is logged? If so where? Checking controller logs was not so successful
  • If not, how can I enable logging? I can see some logs in the code, but not entirely sure how to enable this
  • Could someone point me towards the go file which has the logic for Azure? I was trying to find it but wasn’t so successful

If I could get some help on these points that would be greatly appreciated!

My current workaround is to destroy the controller and recreate it, but obvoiusly that’s not optimal

Peter

1 Like
#2

Try run:

“juju update-credentials azure”

Then select “update controller only” if you have your credentials on the client.

This will likely trigger the juju controller to log you back in to azure and the message might go away.

This happens to us all the time when using vsphere aswell.

#3

When Juju makes an api call to the cloud and an error is returned, it will perform a check to see if the root cause of the error is due to a permission / credential issue. If that is the case, the credential is marked as invalid and any models using that credential are suspended. This is to avoid Juju unnecessarily performing operations that would be known to fail if the credential expires or whatever.

The actual credential checks are dependent on the cloud being used. The Azure checks are in the MaybeInvalidateCredential() method in provider/azure/internal/errorutils/error.go. The check is simply whether the http status code is one of

http.StatusUnauthorized
http.StatusPaymentRequired
http.StatusForbidden
http.StatusProxyAuthRequired

I am pretty sure the actual error itself will be logged in the model logs but maybe not whether the credential is marked as invalid or not. But you should be able to see the logged root cause error and look if it has one of the above status codes.

Out of the box, Juju will set up an Azure credential by creating a service principal off the “Juju” Enterprise Application in the canonical.com Active Directory (app id is 60a04dc9-1857-425f-8076-5ba81ca53d66). This app has all the permissions necessary for Juju to operate. So you could compare whether your custom permissions match what Juju requires.

At a a guess, Juju is trying to perform a cloud operation for which the service principal does not grant it permission and so the credential is getting invalidated.

As Erik suggests, once the permission issue is fixed, you can trigger a re-evaluation of the credential validity by updating the credential.

#4

Thanks both!

Unfotunatlely update credentials did not work, but I will try compare the pre-built SP to what i’m working with here and report back with the cause.

#5

@pjds

Can you paste the output or ideally create a video with ascinema https://asciinema.org/

#6

I had a similar thing with MaaS once
I have my doubts about juju update-credentials.

Any possibility you put a resource lock on the resource group?
I believe my issue happened because I locked the machine in MaaS.

Check out this post

@anastasia-macmood helped me out and I wrote up what fixed my problem, maybe it could you?. :thinking:

#7

hey ya’ll thanks for the input - I was able to narrow down the issue to a missing permission Microsoft.Network/virtualNetworks/subnet/read. Thanks for the help!