The goal of this tutorial is to provide some practical information for managing credentials. It tries to not reiterate what is already provided on the Credentials page but approaches the subject from a more practical level where often misunderstood concepts are reinforced and field-reported scenarios are looked at.
The contents of this tutorial was presented on the Juju Show (April 10 2019), in abridged form, with a slide deck. PDF available here: Lets-talk-about-credentials.pdf (238.7 KB).
Prerequisites
The following prerequisites are assumed as a starting point for this tutorial:
- You’re using Ubuntu 18.04 LTS (Bionic).
- Juju (
v.2.5.2
) is installed. See the Installing Juju page. - You have chosen a backing cloud and have created a controller for it. Refer to the Clouds page to get started with a cloud and controller.
This guide uses Google GCE as a backing cloud.
Credential operations
This is the list of the discreet operations that are possible when managing credentials:
- Add a local credential
- Remove a local credential
- List local credentials
- List remote credentials
- Set a default local credential
- Relate a remote credential to a model
- Change a local credential
- Change a remote credential
There are all covered on the [Credentials][credentials] page.
Topics covered in this tutorial
- Credential definition
- Local credentials
- Default local credential
- Remote credentials
- Credential used when creating a controller
- Credential used when adding a model
- Tracking what remote credential is used by a model
- Managing credentials with multiple users
- Dealing with expired credentials
- Dealing with a reanimated cloud account
- Dealing with inert credentials
Credential definition
A Juju credential represents a collection of authentication material (like username & password, or client id & secret key) that is specific to a cloud and to a Juju user. This bundle of data is managed by means of a user-defined string: the credential name. This name is not a Juju user nor is it the username of the underlying cloud account. It is an arbitrary label that is assigned when the credential was created with the add-credential
command.
Every credential is born (created) from a specific Juju client, which, in turn, is bound to an independent computer host (“device”). There is thus no central way to constrain what credential names get used nor oversee what authentication material they represent. Because of this, different devices can associate the same name with different material, or the opposite, a different name with the same material. This is something to keep in mind, especially in a multi-user context.
Local credentials
When a credential is created it is local to the Juju client that created it. It is therefore known as a local credential and can be listed with the credentials
command. Specifying YAML formatted output makes things explicit:
juju credentials --format yaml
Sample output:
local-credentials:
google:
gandalf:
auth-type: oauth2
client-email: 524443925537-compute@developer.gserviceaccount.com
client-id: "112674610382273922048"
project-id: juju-gce-1725
Notice how the output says ‘local-credentials’.
Here, ‘gandalf’ is the credential name for cloud ‘google’ for the currently logged in Juju user.
Default local credential
When a Juju operation is in need of a local credential it is either specified implicitly, via a defined default local credential, or explicitly, via the --credential
option.
A credential can be defined as the default local credential in two ways:
- Manually by using the
set-default-credential
command - Automatically when the
bootstrap
command is invoked
The automatic method is applied when a sole local credential exists when a controller is created.
The default local credential is annotated with an asterisk in the output for the credentials
command. Below, credential ‘tharkun’ is the default local credential for cloud ‘google’ for the currently logged in Juju user:
Cloud Credentials
google tharkun*, gandalf, mithrandir
Remember:
The --credential
option always refers to a local credential.
Remote credentials
In order for Juju to make use of a cloud it must first authenticate with that cloud, and does so via the Juju controller. Therefore, a local credential must first get uploaded from the client to the controller. Once this happens the credential become known as a remote credential. The show-credentials
command lists all remote credentials for the currently logged in Juju user:
juju show-credentials
Sample output:
controller-credentials:
google:
gandalf:
content:
auth-type: oauth2
client-email: 524443925537-compute@developer.gserviceaccount.com
client-id: "112674610382273922048"
project-id: juju-gce-1725
models:
controller: admin
default: admin
Notice how the output says ‘controller-credentials’.
Here we see that credential ‘gandalf’ is associated with cloud ‘google’ and two models: ‘controller’ and ‘default’. The currently logged in user (the owner of credential ‘gandalf’) is shown to have ‘admin’ model access to both of the aforementioned models.
A remote credential is required whenever a model is created. There are two cases:
- a model is created implicitly via the
bootstrap
command - a model is created with the
add-model
command
In the first case a local credential is solicited to become a remote credential and in the second case either a local credential or an existing remote credential is solicited.
Recall:
Besides creating a new controller, the bootstrap
command also sets up models ‘controller’ and ‘default’.
A remote credential is associated with one cloud, one Juju user, and one or more models.
Note that it is possible for a remote credential to not be related to a model (the model was removed or a private cloud, although very rare, may not require authentication).
Since v.2.6.0
, a credential needs to be added to a cloud before adding that cloud to an existing (multi-cloud) controller.
Credential used when creating a controller
When a controller is created a local credential is uploaded to the controller and related to models controller
and default
. The local credential that gets used is selected according to the following rules, in the order given:
- the one that has been explicitly chosen via the
--credential
option - the one that has been defined as the default local credential
- the sole existing one
The bootstrap
command will error out if a credential cannot be found using the above rules. This would occur when the following is true: there are multiple local credentials; the --credential
option was not used; and a default local credential was not set.
Remember: Only local credentials are considered when creating a controller.
Credential used when adding a model
When a model is added a remote credential is related to that model. The remote credential that gets used (for the current Juju user and the given cloud) is selected according to the following rules, in the order given:
- an uploaded local credential, chosen via the
--credential
option - the sole existing one
Recall that the --credential
option always refers to a local credential. Therefore, in this context, the local credential is first uploaded to the controller, thereby becoming a remote credential, and then related to the model.
The add-model
command will error out if a credential cannot be found using the above rules. This would occur when the following is true: there are multiple remote credentials and the --credential
option was not used.
Note: There is no concept of an actual default remote credential.
Tracking what remote credential is related to a model
To determine what remote credential is related to a model examine the output of the show-model
command. It will include the model’s credential name, the credential owner (the Juju user who uploaded it), and the cloud:
juju show-model default
Sample output:
default:
name: admin/default
short-name: default
.
.
.
credential:
name: jlaurin
owner: admin
cloud: aws
Above, remote credential ‘jlaurin’ is related to model ‘default’.
As we saw earlier, the show-credentials
command can be used to determine what model (or models) a credential is related to:
juju show-credentials
Sample output:
controller-credentials:
google:
saruman:
content:
.
.
.
models:
isengard: admin
orthanc: admin
Above, remote credential ‘saruman’ is related to models ‘isengard’ and ‘orthanc’.
If there were multiple remote credentials they would have all been shown. To target a single cloud:credential combination, use the show-credential
command:
juju show-credential google saruman
Managing credentials with multiple users
Let’s set up a multi-user context by adding a user and granting that user ‘add-model’ permissions:
juju add-user frodo
juju grant frodo add-model
Once this user has registered the controller (register
command) that user may attempt to add a model:
juju add-model shire
This will generate a message indicating that a credential has not been found. This is due to user ‘frodo’ being devoid of both a remote credential and a local credential:
ERROR detecting credentials for "google" cloud provider: gce credentials not
found
If multiple local credentials are added, say ‘potatoes’ and ‘beer’, and the operation is re-attempted an error is once again emitted. This time it’s due to not having selected a credential from among the multiple ones available:
ERROR more than one credential is available. List credentials with:
juju credentials
and then run the add-model command again with the --credential option.
This user can therefore add a model in this way:
juju add-model --credential potatoes shire
Dealing with expired credentials
There have been reports of cloud vendors expiring account credentials. The effect of an expired credential is the inability for Juju to administer any models associated with that credential. Note that workloads will continue to run.
To rectify the situation, first contact the cloud vendor and update your existing credential or re-issue a new one. Once that’s done, you will need to do one of the following:
- update the existing Juju credential on the controller
- create a new credential and relate it to affected models
We’ll look at each of these options in greater detail below.
You can simulate an expired credential by deactivating the corresponding account on your cloud’s dashboard or via its API.
Option 1: Update the existing remote credential
To change the remote credential the update-credential
command is used. It does this by uploading the identically named local credential to the controller. Therefore, the local credential first needs to be changed.
The local credential contents are changed with the add-credential
command, where the --replace
option is needed because the credential is already existing.
juju add-credential --replace google -f credentials-gandalf-changed.yaml
juju update-credential google gandalf
Here, the contents of credential ‘gandalf’ was modified in file credentials-gandalf-changed.yaml
.
In v.2.6.0
, the update-credential
command supports the consumption of a YAML file. So here we bypass the local client cache:
juju update-credential google gandalf -f credentials-gandalf-changed.yaml
Option 2: Create a new credential and related it to affected models
To add an entirely new local credential the add-credential
command is used. To relate a remote credential to a model the set-credential
command is used. Here we presume that affected models are ‘shire’ and ‘rohan’:
juju add-credential google -f credentials-mithrandir.yaml
juju set-credential -m shire google mithrandir
juju set-credential -m rohan google mithrandir
Above, the contents for remote credential ‘mithrandir’ were added to file credentials-mithrandir.yaml
.
A credential targeted with the set-credential
command will upload the identically-named local credential if it is not found remotely.
Dealing with a reanimated cloud account
Closely related to the previous case of credentials expired by the cloud vendor is the scenario where the cloud account was deactivated by the cloud account administrator. Every cloud vendor has the functionality to do this and it can be accomplished via the cloud’s API or via its web dashboard.
A deactivated cloud account will naturally also result in an invalid credential on the Juju side. A reasonable way forward would be to simply reactivate the account, but this is insufficient. This is because Juju has already invalidated the credential. The solution is as before: “update” the existing Juju credential on the controller with the update-credential
command. However, you do not need to actually replace the credential’s contents. You just need to announce to the controller that the credential is a good one.
Assuming, therefore, that the account has been made active again and it is associated with Juju credential ‘tharkun’, simply proceed as follows:
juju update-credential google tharkun
Dealing with inert credentials
Closely related to the previous case of credentials not working as a result of a deactivated cloud account is the case where the cloud account credentials inexplicably stop working. This has been reported with vSphere, for instance, and whose cause is probably due to a glitch in the vendor software.
The solution is as before: “update” the existing Juju credential on the controller with the update-credential
command. However, you do not need to actually replace the credential’s contents. You just need to announce to the controller that the credential is a good one.
Assuming, therefore, that the account is associated with Juju credential ‘goblins’, simply proceed as follows:
juju update-credential vsphere goblins