Cannot Grant Candid User Access to Controller Model

juju
1

Our original juju controller is on 3.5.5 and is integrated with Candid, allowing us to grant access to models via LDAP.

To upgrade to 3.6.1 we backed up our 3.5.5 controller, bootstrapped a new 3.6.1 controller, then restored the backup onto the new controller:

juju create-backup -m oldcontroller:controller
juju bootstrap cloudname/default newcontroller --bootstrap-series=jammy --bootstrap-constraints "tags=juju" --config identity-url=IdentityUrlValue --config identity-public-key=IdentityPublicKeyValue --config allow-model-access=true
juju switch newcontroller:controller
juju scp juju-restore 0:
juju scp JujuBackupFile.tar.gz 0:
juju ssh 0
./juju-restore --copy-controller JujuBackupFile.tar.gz

After doing so we are unable to grant LDAP users access to the controller model:

ERROR could not look up model access for user: state changing too quickly; try again soon

The error message suggests this may be transient, but it is not.

We are able to grant LDAP users access to other (non-controller) models. I was able to grant an LDAP user access to the controller model after the bootstrap, but before restoring the backup.

Has anyone encountered this scenario before or have any suggestions on how to resolve it?

2

Hey @GrumpyBear, when you do juju create-backup juju will literally create a backup of its Mongo state. When you load that onto a controller of a different version this can cause issues, if any of the mongo database keys have changed between versions the old data will appear missing to the 3.6.1 controller. Our documentation on this could admittedly be clearer.

The error you are getting, state changing too quickly comes from our mongo driver. The error occours when a precondition for a query fails (despite the name). I would guess that there is some query that is expecting a collection to exist, or some field within it, that cannot be found due to the fact that you have a 3.5.5 state loaded on a 3.6.1 controller.

To configure integration with Candid on the 3.6.1 controller you will have to follow the same procedure you used to initially configure it on the 3.5.5 controller. If there are any issues you have with that let me know!

3

Hi @aflynn - Thanks for the response. I used the juju create-backup / juju-restore process as that is what I found outlined in the documentation here: https://juju.is/docs/juju/manage-controllers#upgrade-your-controllers-minor-or-major-version

Is there a different process I should be following to port the old controller configuration over to the new controller? I did include the Candid information while bootstrapping the new controller and that seemed to be fine before restoring the backup onto it.

4

Hello,

I have similar issue after migration of Juju controller 3.1 → 3.6 (same process as described above) but removing and creating the user doesn’t work:

ubuntu@jump:~$ juju show-user prometheus
user-name: prometheus
access: login
date-created: 8 hours ago
last-connection: 28 seconds ago
ubuntu@jump:~$ juju remove-user prometheus
WARNING! This command will permanently archive the user "prometheus" on the "juju36"
controller. This action is irreversible and you WILL NOT be able to reuse
username "prometheus".

If you wish to temporarily disable the user please use the `juju disable-user`
command. See
 `juju help disable-user` for more details.

Continue (y/N)? y
User "prometheus" removed
ubuntu@jump:~$ juju add-user prometheus
User "prometheus" added
Please send this command to prometheus:
    juju register REDACTED

"prometheus" has not been granted access to any models. You can use "juju grant" to grant access.
ubuntu@jump:~$ juju grant prometheus read controller -c juju36
ERROR could not look up model access for user: state changing too quickly; try again soon
ubuntu@jump:~$ juju show-user prometheus -c juju36
user-name: prometheus
access: login
date-created: 1 minute ago
last-connection: 14 seconds ago
ubuntu@jump:~$ juju grant prometheus read controller -c juju36 --debug
22:06:38 INFO  juju.cmd supercommand.go:56 running juju [3.6.2 87cae7505aee356eda90d98ae345e1c11eb26c72 gc go1.23.4]
22:06:38 DEBUG juju.cmd supercommand.go:57   args: []string{"/snap/juju/29493/bin/juju", "grant", "prometheus", "read", "controller", "-c", "juju36", "--debug"}
22:06:38 INFO  juju.juju api.go:86 connecting to API addresses: [100.112.114.139:17070]
22:06:38 DEBUG juju.api apiclient.go:1035 successfully dialed "wss://100.112.114.139:17070/api"
22:06:38 INFO  juju.api apiclient.go:570 connection established to "wss://100.112.114.139:17070/api"
22:06:39 DEBUG juju.api monitor.go:35 RPC connection died
ERROR could not look up model access for user: state changing too quickly; try again soon
22:06:39 DEBUG cmd supercommand.go:549 error stack:
github.com/juju/juju/rpc/params.ErrorResults.Combine:91: could not look up model access for user: state changing too quickly; try again soon
ubuntu@jump:~$ juju users
Controller: juju36

Name        Display name  Access     Date created   Last connection
admin*      admin         superuser  2025-02-19     just now
baruser                   superuser  2024-06-13     never connected
prometheus                login      4 minutes ago  22 seconds ago