The aim of this post is to introduce some new work which will deliver secrets for Juju as a first class part of the Juju model. The work is under heavy development, and is not complete. But there are some key primitives which are usable for those who would like to get hands on with the feature as it evolves.
Warning:
This feaure is subject to change. It can be used for testing but should not be used in production.
Things like secret URL format, CLI syntax and behaviour, hook semantics etc are subject to change.
Background and motivation
Juju currently doesn’t explicitly model the concept of secrets. Thus when a charm needs to be configured to use a credential or token of some sort to gain authenticated access to a restricted API, the secret is passed in directly without any obfuscation or disclosure protection. This also applies to inter-charm exchange of data via relations. The net effect is that Juju has no way to maintain the integrity of workload secrets; these are exposed in various ways, including log lines, reading charm configuration, dumping relation data bags etc.
Another consideration is that government regulatory and/or customer compliance requirements for management of sensitive digital assets (eg GDPR, SOX etc) will mandate that certain practices be followed regarding the storage and controlled access to sensitive digital assets such as secrets. A requirement is often that trusted and/or standards compliant secret access stores or workflows be used; Juju will need to integrate with these as well as providing a fallback implementation for development or other less stringent deployment scenarios.
Third Party Interfaces / Components
A key component of the solution will be a secrets-as-a-service backend used to manage access to the secrets, eg we already have the capability to provision a charm based Vault backend.
On Kubernetes, we can make use of Secrets Resources but note these store sensitive information as simply base64 encoded strings by default. However, they do provide an out-of-the-box solution that we can leverage to start with.
Getting started
To gain access to this feature, the use of a feature flag is currently required. This needs to be set up when bootstrapping a new controller.
export JUJU_DEV_FEATURE_FLAGS=secrets
juju bootstrap ...
All of the examples called out below are possible with the current Juju 2.9 candidate releases, but may be subject to change.
Limitations
This is work in progress so several key pieces are not yet available. This includes:
- operator framework APIs
- grant / revoke access to secrets (secret rbac)
- secret lifecycle management
- full workflow for managing secret rotation
- secrets in charm config
Backends: At the time of writing, only the Juju secrets backend is supported; secrets are stored in Mongo with the Juju model to which they belong.
Key concepts
Identifying a secret
Secrets are currently tied to an application. They are created by a charm and are identified by a secret URL. The secret URL is passed across a relation to a consuming charm which needs to use the secret. Previously, the secret value itself would need to be put into relation data. The consuming charm can request the value of a secret for a given URL and use it as needed when interacting wth the related workload.
Secret values
Secrets themselves are data bag of key of values. There’s currently no strong typing of different secret kinds. Many secrets will consist of just a single value, like a password or an API token. Other secrets might comprise multiple values, like a TLS certificate and private key. There’s some synatctic sugar which is used for dealing with single value secrets which we’ll see later.
Mediating access to a secret
The charm which creates a secret can also grant and revoke access to that secret. Currently, access is scoped to another application; a charm will grant access to that application when a relation is created, and any unit of that application may get the secret value. There’s plans to restrict access to individual units, but to do this cleanly requires discussing changes to the relation data model.
An access grant may also include a relation id. This ties the grant to the relation so that when the relation is removed, so is the access grant.
Rotating a secret
Secrets may be created with a specified rotation interval, eg X hours or Y days. Each time a secret is rotated, a new revision of that secret is stored. The default is to always get the latest active revision of a secret; previous revisions may still be accessed by including the revision number in the request.
Staging a secret update
Secrets may be updated but marked as “staged” / “pending”. Such secrets get a new revision number but are not served as the value of a secret until marked as “active”. This is used as part of the workflow to roll out updates when a secret is rotated.
Secret Identitifiers
Secrets are identitied by a secret URL with scheme secret. They are safe to view, log, exchange between charms etc - access to the actual secret payload is mediated via Juju and the access rules in place
Secret URLs may optionally specify individual attributes to fetch, as well a particular revision.
The format of a secret URL is currently:
secret://app/<secret-path>[/<rev>][#<key>]
or
secret://id/<id_number>
Each secret gets a sequence id ; ids are unique to a model.
Examples:
secret://app/gitlab/apitoken
(used to specify the latest revision of the secret)
secret://app/proxy#key
(used to specify to "key" attribute of the secret)
secret://app/proxy/5
(used to specify revision 5 of the secret)
secret://app/proxy/5#cert
(used to specify the "cert" attribute of revision 5 of the secret)
Secret Structure
Secrets are modelled as a map of key value pairs. This allows for secrets which require more than one attribute, like a TLS certificate and key. It is expected that most of the time, a given secret will only require the one attribute (like a password or encryption key or api token). We’ll optimise the UX to nicely handle this common case; a single item map with a key named data represents a singular secret value. Dotted notation is used for nested values (see SecretD below).
Consider an application “mariadb” with secret data bags defined as follows:
SecretA
data=s3cret!
SecretB
cert=somecert
SecretC
cert=somecert
key=somekey
SecretD
foo.bar=1234
foo.baz=5678
hello=world
The following examples show how the secrets might be accessed via their URL:
get-secret secret://app/mariadb/SecretA
-> s3cret!
get-secret secret://app/mariadb/SecretA#data
-> s3cret!
get-secret secret://app/mariadb/SecretB#cert
-> somecert
get-secret secret://app/mariadb/SecretB
-> {"cert":"somecert"}
get-secret secret://app/mariadb/SecretC#cert
-> somecert
get-secret secret://app/mariadb/SecretC
-> {"cert":"somecert", "key":"somekey"}
get-secret secret://app/mariadb/SecretD
-> {"hello":"world", "foo": {"bar":"1234", "baz":"5678"}}
get-secret secret://app/mariadb/SecretD#foo
-> {"bar":"1234", "baz":"5678"}
Secret Metadata
Secrets are maintained with metadata including:
- secret id
- status (active, staged)
- description
- rotate interval
- last rotate timestamp
- tags (arbitrary key values)
- created / updated timestamp
- access rules
Charms which create secrets may update the following metadata items:
- description
- tags
- rotate interval
- status
Tags are meaningful to the charm and could be used to store relevant state about the secret.
Secret Lifecycle
Secrets created by a charm are tied to the lifecycle of the charm’s application. When the application is removed, so too are any associated secrets.
Any grants scoped to a related application or unit are removed when any of the related entities are removed. In the case of unit scoped grants, this is done as the related unit leaves the relation scope as part of its departing relation workflow. For grants which are also tied to a specific relation, the grant is removed at the same time as the relation.
Overview of secrets and charms
This section summarises the key primitives used by charms for interacting with secrets. Subsequent sections provide examples of how they are used.
Managing secrets
Not yet implemented: Operator framework APIs will be used to create and manage secrets. These are not yet available, so direct use of hook commands is needed.
The currently implemented hook commands are:
create-secret <name>
[--base64]
[--staged]
[--rotate <duration>]
[--tag <key>=<value>...]
[--description <sometext>]
[<key>=<value>...]
secret-update <name>
[--base64]
[--active | --staged]
[--rotate=<duration>]
[--tag <key>=<value>...]
[--description <sometext>]
[<key>=<value>...]
secret-get <secret-id>
[--base64]
Still todo are:
- secret-metadata
- secret-delete
- secret-grant
- secret-revoke
Secret hooks
Hooks are used to inform a charm about events relevant to secrets the charm has created.
The hooks include:
- secret-rotated
- secret-changed
secret-rotated is fired when it is time to rotate a secret.
secret-changed is fired when a secret’s status changes.
Example: creating and accessing secrets
Secret access control is not yet implemented - this section illustrates a simple usage scenario where a database charm might generate a password secret and share that with a related blog charm.
In the examples, we’ll exec the hook commands directly since the APIs are still being implemented in the operator framework.
The Python boilerplate for running the hook commands will be ommited, ie
...
pw = 's3cr3t!'
subprocess.check_output(['secret-create', 'password', pw])).strip()
...
will be written as
secret-create password s3cr3t!
First we deploy some charm and relate them:
juju deploy mariadb
juju deploy mediawiki
juju relate mediawiki:db mariadb
In the relation-joined hook for mariadb, we want to generate a password to hand to mediawiki.
secret-create password --tag hello=world --description "Password for mariadb" s3cr3t!
-> secret://app/mariadb/password/1
Upon success, the secret URL is printed to stdout. The secret URL can be put into relation data:
relation-set --app dbpass=secret://app/mariadb/password/1
On the mediawiki side, the value of the secret can be obtained from the relation-changed event:
secret_url=$(`which relation-get` --app dbpass)
secret-get $secret_url
-> s3cr3t!
Base 64 data
Secret data is base 64 encoded when stored. To create a secret with data that is already base 64 encoded, use the --base64 flag.
Similarly, to access the “raw” base 64 data when getting a secret value, again use the --base64 flag.
secret-create foo hello
-> secret://app/mariadb/foo/1
secret-get secret://app/mariadb/foo/1 --base64
-> aGVsbG8=
secret-create --base64 bar aGVsbG8=
-> secret://app/mariadb/bar/1
secret-get secret://app/mariadb/bar/1
-> hello
Juju Secrets CLI
The Juju secrets command can be used to inspect secrets which have been created, including optionally the secret value. There’s no access control yet; this is useful primarily for debugging at this stage.
The default output is a tabular summary of all secrets:
$ juju secrets
ID Revision Rotate Backend Path Age
1 1 never juju app/mariadb/password 38 minutes ago
2 1 never juju app/mariadb/foo 8 minutes ago
3 1 never juju app/mariadb/bar 7 minutes ago
Using YAML or JSON, you can see more detail, as well as optionally the secret values using --show-secrets
$ juju secrets --format yaml --show-secrets
- ID: 1
URL: secret://app/mariadb/password
revision: 1
path: app/mariadb/password
status: active
version: 1
description: Password for mariadb
tags:
hello: world
backend: juju
create-time: 2021-09-28T05:15:57Z
update-time: 2021-09-28T05:15:57Z
value:
data: s3cr3t!
- ID: 2
URL: secret://app/mariadb/foo
revision: 1
path: app/mariadb/foo
status: active
version: 1
backend: juju
create-time: 2021-09-28T05:45:18Z
update-time: 2021-09-28T05:45:18Z
value:
data: hello
- ID: 3
URL: secret://app/mariadb/bar
revision: 1
path: app/mariadb/bar
status: staged
version: 1
backend: juju
create-time: 2021-09-28T05:46:23Z
update-time: 2021-09-28T05:46:23Z
value:
data: hello
Creating secrets that need to be rotated
If a secret should be rotated every so often, specify the rotation interval using the --rotate option:
secret-create password --rotate=12h
-> secret://app/mariadb/password/1
Supported durations include days (d) or hours (h).
A rotate value of 0 disables rotation, eg
secret-update password --rotate=0
Rotating secrets
Ideas only: The implementation for secret rotation is still at the ideas stage, but is included here to share some current thinking. Not in scope currently are secrets backends which support automatic secret rotation.
A secret may need to be rotated every so often. In some cases, the charm will need to fetch a new value for the secret from an external API (eg Openstack fernet key). In other cases, the charm may itself simply need to re-generate a new value for the secret. Either way, 2 hooks are used by Juju in the rotation workflow:
-
secret-rotatewhen it is time to rotate the secret -
secret-changedif the secret’s status changes
The status of a secret can be used to assist in managing the workflow. This workflow is similar to options supported by other systems like AWS Secrets Manager to name one example.
This approach offers simplicity but it means there will commonly be a brief time period where a client might still attempt to use a stale secret - clients need to use a backoff/retry strategy plus jitter, and this will mitigate any resulting client downtime. The trade off seems reasonable.
Passwords: Under consideration is support for “password” secrets, where Juju itself will generate a new secret value based on complexity criteria supplied by the charm when the secret is created.
Consider the case of a mariadb charm needing to update the database password.
The sequencing of the hooks below allows the unit agent to be interrupted at any point and the charm can continue the workflow when the agent comes back online and re-runs any uncommitted hook. This is why the “secret-changed” hook is used to inform the charm that the secret status has been updated in the Juju model and allows the generation of the new secret value to be done separately from the workload update and change propagation.
The general sequence of events is:
secret-rotate hook (time to rotate)
-> charm generates new secret, stores as staged
secret-changed hook (secret is staged)
-> charm updates workload to use new password
-> charm sets secret to be active
secret-changed hook (secret is active)
-> charm notifies related units the secret has changed
juju fires secret-rotate hook
-> outcome is to generate new secret and store in model as staged
hook context contains:
- secret URL
- secret tags
(tags are set by the charm when the secret is created and may be updated after that; they are included in the rotate hook as a means of avoiding the charm having to maintain state about the secret).
Charm updates the secret to store a new password but sets the secret as “staged”.
Existing clients asking for the latest secret still see the previous active one and can still access mariadb database at this stage using that password
new_password=<charm generates new secret password>
secret-update $JUJU_SECRET_ID password=$new_password --staged
-> secret://app/madiadb/password/666
juju fires secret-changed hook
-> outcome is to update workload to use new secret and update as active
hook context contains:
- secret URL plus revision
- secret tags
- secret-status=staged
Charm updates the mariadb workload password to the new secret value
at this point, there’s a window where mariadb clients will fail to access the database
Charm makes the new secret status “active”.
This results in any requests to get the latest secret getting the new value.
secret-update $JUJU_SECRET_ID --active
-> secret://app/madiadb/password/666
juju fires secret-changed hook
-> outcome is to notify related charms that secret has changed
hook context contains:
- secret identifier plus revision
- secret tags
- secret-status=active
Charm notifies related charms that the secret has changed.
If related charms always do a secret-get at the time they use a secret (rather than caching it), any potential downtime due to the rotation is minimised, and the notification becomes more informative than critical.
The point about clients avoiding (where possible) caching the secret is worthy to call out - if a secret is only sporadically fetched to say initiate a long held connection, then always fetching “latest” minimises downtime when a secret is updated. If updating the secret causes a client connection to drop, the client would fetch the secret as part of re-establishing the connection and things would proceed as normal.
Updating a secret just changes the secret content, not the name. The secret revision is updated and this is included in the secret URL printed on success.
The charm may then update relation data with the new secret URL to inform consumers of the secret it has changed.
Note:
It is anticipated that the secret-metata hook command might be used to inform the charm of which relations should be updated with the revised secret URL for the rotated secret.
