How to add secrets to a charm

doc
, ,
1

See first: Juju | Secret

See also: Secret events

In the context of the charm SDK, a secret always refers to a charm-created secret.

This feature is available starting with ops 2.0.0, but only when using Juju 3.0.2 or greater.

As of version 3.0, Juju supports secrets. Secrets allow charms to securely exchange data via a channel mediated by the Juju controller. In this document we will guide you through how to use the new secrets API as exposed by the Ops library.

We are going to assume:

  • Surface knowledge of the ops library
  • Familiarity with secrets terminology and concepts (see Secret events)

Contents:

Setup

Let us assume that you have two charms that need to exchange a secret. For example, a web server and a database. To access the database, the web server needs a username/password pair. Without secrets, charmers were forced to exchange the credentials in clear text via relation data, or manually add a layer of (presumably public key) encryption to avoid broadcasting that data (because, remember, anyone with the Juju CLI and access to the controller can run juju show-unit).

In secrets terminology, we call the database the owner of a secret, and the web server its observer.

Starting from the owner, let’s see what changes are necessary to the codebase to switch from using plain-text relation data to one backed by Juju secrets.

Owner: Add a secret

Presumably, the owner code (before using secrets) looked something like this:

class MyDatabaseCharm(ops.CharmBase):
    def __init__(self, *args, **kwargs):
        ...  # other setup
        self.framework.observe(self.on.database_relation_joined, 
                               self._on_database_relation_joined)

    ...  # other methods and event handlers
   
    def _on_database_relation_joined(self, event: ops.RelationJoinedEvent):
        event.relation.data[self.app]['username'] = 'admin' 
        event.relation.data[self.app]['password'] = 'admin'  # don't do this at home

Using the new secrets API, this can be rewritten as:

class MyDatabaseCharm(ops.CharmBase):
    def __init__(self, *args, **kwargs):
        ...  # other setup
        self.framework.observe(self.on.database_relation_joined,
                               self._on_database_relation_joined)

    ...  # other methods and event handlers

    def _on_database_relation_joined(self, event: ops.RelationJoinedEvent):
        content = {
            'username': 'admin',
            'password': 'admin',
        }
        secret = self.app.add_secret(content)
        secret.grant(event.relation)
        event.relation.data[self.app]['secret-id'] = secret.id

Note that:

  • We call add_secret on self.app (the application). That is because we want the secret to be owned by this application, not by this unit. If we wanted to create a secret owned by the unit, we’d call self.unit.add_secret instead.
  • The only data shared in plain text is the secret ID (a locator URI). The secret ID can be publicly shared. Juju will ensure that only remote apps/units to which the secret has explicitly been granted by the owner will be able to fetch the actual secret payload from that ID.
  • The secret needs to be granted to a remote entity (app or unit), and that always goes via a relation instance. By passing a relation to grant (in this case the event’s relation), we are explicitly declaring the scope of the secret – its lifetime will be bound to that of this relation instance.

Observer: Get a secret

Before secrets, the code in the secret-observer charm may have looked something like this:

class MyWebserverCharm(ops.CharmBase):
    def __init__(self, *args, **kwargs):
        ...  # other setup
        self.framework.observe(self.on.database_relation_changed,
                               self._on_database_relation_changed)

    ...  # other methods and event handlers

    def _on_database_relation_changed(self, event: ops.RelationChangedEvent):
        username = event.relation.data[event.app]['username']
        password = event.relation.data[event.app]['password']
        self._configure_db_credentials(username, password)

With the secrets API, the code would become:

class MyWebserverCharm(ops.CharmBase):
    def __init__(self, *args, **kwargs):
        ...  # other setup
        self.framework.observe(self.on.database_relation_changed,
                               self._on_database_relation_changed)

    ...  # other methods and event handlers

    def _on_database_relation_changed(self, event: ops.RelationChangedEvent):
        secret_id = event.relation.data[event.app]['secret-id']
        secret = self.model.get_secret(id=secret_id)
        content = secret.get_content()
        self._configure_db_credentials(content['username'], content['password'])

Note that:

  • The observer charm gets a secret via the model (not its app/unit). Because it’s the owner who decides who the secret is granted to, the ownership of a secret is not an observer concern. The observer code can rightfully assume that, so long as a secret ID is shared with it, the owner has taken care to grant and scope the secret in such a way that the observer has the rights to inspect its contents.
  • The charm first gets the secret object from the model, then gets the secret’s content (a dict) and accesses individual attributes via the dict’s items.

Manage secret revisions

If your application never needs to rotate secrets, then this would be enough. However, typically you want to rotate a secret periodically to contain the damage from a leak, or to avoid giving hackers too much time to break the encryption.

Creating new secret revisions is an owner concern. First we will look at how to create a new revision (regardless of when a charm decides to do so) and how to observe it. Then, we will look at the built-in mechanisms to facilitate periodic secret rotation.

Owner: Create a new revision

To create a new revision, the owner charm must call secret.set_content and pass in the new payload:

class MyDatabaseCharm(ops.CharmBase):

    ... # as before

    def _rotate_webserver_secret(self, secret):
        content = secret.get_content()
        secret.set_content({
            'username': content['username'],              # keep the same username
            'password': _generate_new_secure_password(),  # something stronger than 'admin'
        })

This will inform Juju that a new revision is available, and Juju will inform all observers tracking older revisions that a new one is available, by means of a secret-changed hook.

Observer: Update to a new revision

To update to a new revision, the web server charm will typically subscribe to the secret-changed event and call get_content with the “refresh” argument set (refresh asks Juju to start tracking the latest revision for this observer).

class MyWebserverCharm(ops.CharmBase):
    def __init__(self, *args, **kwargs):
        ...  # other setup
        self.framework.observe(self.on.secret_changed,
                               self._on_secret_changed)

    ...  # as before

    def _on_secret_changed(self, event: ops.SecretChangedEvent):
        content = event.secret.get_content(refresh=True)
        self._configure_db_credentials(content['username'], content['password'])

Peek at a secret’s payload

Sometimes, before reconfiguring to use a new credential revision, the observer charm may want to peek at its contents (for example, to ensure that they are valid). Use peek_content for that:

    def _on_secret_changed(self, event: ops.SecretChangedEvent):
        content = event.secret.peek_content()
        if not self._valid_password(content.get('password')):
           logger.warning('Invalid credentials! Not updating to new revision.')
           return
        content = event.secret.get_content(refresh=True)
        ...

Label secrets

Sometimes a charm will observe multiple secrets. In the secret-changed event handler above, you might ask yourself: How do I know which secret has changed? The answer lies with secret labels: a label is a charm-local name that you can assign to a secret. Let’s go through the following code:

class MyWebserverCharm(ops.CharmBase):

    ...  # as before

    def _on_database_relation_changed(self, event: ops.RelationChangedEvent):
        secret_id = event.relation.data[event.app]['secret-id']
        secret = self.model.get_secret(id=secret_id, label='database-secret')
        content = secret.get_content()
        self._configure_db_credentials(content['username'], content['password'])

    def _on_secret_changed(self, event: ops.SecretChangedEvent):
        if event.secret.label == 'database-secret':
            content = event.secret.get_content(refresh=True)
            self._configure_db_credentials(content['username'], content['password'])
        elif event.secret.label == 'my-other-secret':
            self._handle_other_secret_changed(event.secret)
        else:
            pass  # ignore other labels (or log a warning)

As shown above, when the web server charm calls get_secret it can specify an observer-specific label for that secret; Juju will attach this label to the secret at that point. Normally get_secret is called for the first time in a relation-changed event; the label is applied then, and subsequently used in a secret-changed event.

Labels are unique to the charm (the observer in this case): if you attempt to attach a label to two different secrets from the same application (whether it’s the on the observer side or the owner side) and give them the same label, the framework will raise a ModelError.

Whenever a charm receives an event concerning a secret for which it has set a label, the label will be present on the secret object exposed by the framework.

The owner of the secret can do the same. When a secret is added, you can specify a label for the newly-created secret:

class MyDatabaseCharm(ops.CharmBase):

    ...  # as before

    def _on_database_relation_joined(self, event: ops.RelationJoinedEvent):
        content = {
            'username': 'admin',
            'password': 'admin',
        }
        secret = self.app.add_secret(content, label='secret-for-webserver-app')
        secret.grant(event.relation)
        event.relation.data[event.unit]['secret-id'] = secret.id

If a secret has been labelled in this way, the charm can retrieve the secret object at any time by calling get_secret with the “label” argument. This way, a charm can perform any secret management operation even if all it knows is the label. The secret ID is normally only used to exchange a reference to the secret between applications. Within a single application, all you need is the secret label.

So, having labelled the secret on creation, the database charm could add a new revision as follows:

    def _rotate_webserver_secret(self):
        secret = self.model.get_secret(label='secret-for-webserver-app')
        secret.set_content(...)  # pass a new revision payload, as before

When to use labels

When should you use labels? A label is basically the secret’s name (local to the charm), so whenever a charm has, or is observing, multiple secrets you should label them. This allows you to distinguish between secrets, for example, in the SecretChangedEvent shown above.

Most charms that use secrets have a fixed number of secrets each with a specific meaning, so the charm author should give them meaningful labels like database-credential, tls-cert, and so on. Think of these as “pets” with names.

In rare cases, however, a charm will have a set of secrets all with the same meaning: for example, a set of TLS certificates that are all equally valid. In this case it doesn’t make sense to label them – think of them as “cattle”. To distinguish between secrets of this kind, you can use the Secret.unique_identifier property, added in ops 2.6.0.

Note that Secret.id, despite the name, is not really a unique ID, but a locator URI. We call this the “secret ID” throughout Juju and in the original secrets specification – it probably should have been called “uri”, but the name stuck.

Owner: Manage rotation

We have seen how an owner can create a new revision and how an observer can update to it (or peek at it). What remains to be seen is: when does the owner decide to rotate a secret?

Juju exposes two separate mechanisms for owner charms to rotate a secret: rotation and expiration. The observer-side mechanism for peeking at or updating to new revisions does not change, so this section will only discuss owner code.

A charm can configure a secret, at creation time, to have one or both of:

  • A rotation policy (weekly, monthly, daily, and so on).
  • An expiration date (for example, in two months from now).

Here is what the code would look like:

class MyDatabaseCharm(ops.CharmBase):
    def __init__(self, *args, **kwargs):
        ...  # other setup
        self.framework.observe(self.on.secret_rotate,
                               self._on_secret_rotate)

    ...  # as before

    def _on_database_relation_joined(self, event: ops.RelationJoinedEvent):
        content = {
            'username': 'admin',
            'password': 'admin',
        }
        secret = self.app.add_secret(content,
            label='secret-for-webserver-app',
            rotate=SecretRotate.DAILY)

    def _on_secret_rotate(self, event: ops.SecretRotateEvent):
        # this will be called once per day.
        if event.secret.label == 'secret-for-webserver-app':
            self._rotate_webserver_secret(event.secret)

Or, for secret expiration:

class MyDatabaseCharm(ops.CharmBase):
    def __init__(self, *args, **kwargs):
        ...  # other setup
        self.framework.observe(self.on.secret_expired,
                               self._on_secret_expired)

    ...  # as before

    def _on_database_relation_joined(self, event: ops.RelationJoinedEvent):
        content = {
            'username': 'admin',
            'password': 'admin',
        }
        secret = self.app.add_secret(content,
            label='secret-for-webserver-app',
            expire=datetime.timedelta(days=42))  # this can also be an absolute datetime

    def _on_secret_expired(self, event: ops.SecretExpiredEvent):
        # this will be called only once, 42 days after the relation-joined event.
        if event.secret.label == 'secret-for-webserver-app':
            self._rotate_webserver_secret(event.secret)

Manage a secret’s end of life

No matter how well you keep them, secrets aren’t forever. If the relation holding the two charms together is removed, the owner charm might want to clean things up and remove the secret as well (the observer won’t be able to access it anyway).

Also, suppose that the owner charm has some config variable that determines who is to be granted access to the db. If that were to change after the charm already has granted access to some remote entity, the database charm will need to revoke access.

Finally, if the owner rotates the charm and all observers update to track the new (latest) revision, the old revision will become dead wood and can be removed.

We show you how to handle all these scenarios below.

Remove a secret

To remove a secret (effectively destroying it for good), the owner needs to call secret.remove_all_revisions. Regardless of the logic leading to the decision of when to remove a secret, the code will look like some variation of the following:

class MyDatabaseCharm(ops.CharmBase):
    ...

    # called from an event handler
    def _remove_webserver_secret(self):
        secret = self.model.get_secret(label='secret-for-webserver-app')
        secret.remove_all_revisions()

After this is called, the observer charm will get a ModelError whenever it attempts to get the secret. In general, the presumption is that the observer charm will take the absence of the relation as indication that the secret is gone as well, and so will not attempt to get it.

Removing a revision

Removing a single secret revision is a more common (and less drastic!) operation than removing all revisions.

Typically, the owner will remove a secret revision when it receives a secret-remove event – that is, when that specific revision is no longer tracked by any observer. If a secret owner did remove a revision while it was still being tracked by observers, they would get a ModelError when they tried to get the secret.

A typical implementation of the secret-remove event would look like:

class MyDatabaseCharm(ops.CharmBase):

    ...  # as before

    def __init__(self, *args, **kwargs):
        ...  # other setup
        self.framework.observe(self.on.secret_remove,
                               self._on_secret_remove)

    def _on_secret_remove(self, event: ops.SecretRemoveEvent):
        # all observers are done with this revision, remove it
        event.secret.remove_revision(event.revision)

Revoke a secret

For whatever reason, the owner of a secret can decide to revoke access to the secret to a remote entity. That is done by calling secret.revoke, and is the inverse of secret.grant.

An example of usage might look like:

class MyDatabaseCharm(ops.CharmBase):

    ...  # as before

    # called from an event handler
    def _revoke_webserver_secret_access(self, relation):
        secret = self.model.get_secret(label='secret-for-webserver-app')
        secret.revoke(relation)

Just like when the owner granted the secret, we need to pass a relation to the revoke call, making it clear what scope this action is to be applied to.

Conclusion

In this guide we have taken a tour of the new ops secrets API. Hopefully you will find that the API is intuitive and provides a clear wrapper around the new juju hook tools. If you find bugs, have suggestions, or want to contribute to the discussion, you can use the tracker on github.

Of course, we have also added facilities in the testing Harness to help charmers test their secrets. This is however a matter for another document.

1 Like
2
3

Thanks for the edit @tmihoc, looking much better now! We might want to align with the rest of the team on this, but I believe the official terminology has evolved: not holder but consumer. After confirming with the leads we’ll need to replace.

1 Like
4

Sounds good – let me know when it’s confirmed!

5

confirmed! We just decided to go to consumer. Will adapt it in the other docs.

1 Like
6

I have a sample charm that tries to implement this, but is currently failing to get secrets via a peer relation.

The error is:

unit-mthaddon-secrets-demo-1: 11:54:01 ERROR unit.mthaddon-secrets-demo/1.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mthaddon-secrets-demo-1/charm/venv/ops/model.py", line 2682, in _run
    result = run(args, **kwargs)
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-mthaddon-secrets-demo-1/secret-get', 'secret:cdlod8ev92qmpavc97i0', '--metadata', '--format=json')' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 48, in <module>
    main(SecretsCharmCharm)
  File "/var/lib/juju/agents/unit-mthaddon-secrets-demo-1/charm/venv/ops/main.py", line 445, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-mthaddon-secrets-demo-1/charm/venv/ops/main.py", line 150, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-mthaddon-secrets-demo-1/charm/venv/ops/framework.py", line 355, in emit
    framework._emit(event)  # noqa
  File "/var/lib/juju/agents/unit-mthaddon-secrets-demo-1/charm/venv/ops/framework.py", line 839, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-mthaddon-secrets-demo-1/charm/venv/ops/framework.py", line 914, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 27, in _on_config_changed
    secret = self.model.get_secret(id=secret_id)
  File "/var/lib/juju/agents/unit-mthaddon-secrets-demo-1/charm/venv/ops/model.py", line 242, in get_secret
    meta = self._backend.secret_get(
  File "/var/lib/juju/agents/unit-mthaddon-secrets-demo-1/charm/venv/ops/model.py", line 2831, in secret_get
    response = self._run(*args, return_output=True, use_json=key is None)
  File "/var/lib/juju/agents/unit-mthaddon-secrets-demo-1/charm/venv/ops/model.py", line 2688, in _run
    raise ModelError(e.stderr)
ops.model.ModelError: b'ERROR secret "cdlod8ev92qmpavc97i0" not found\n'
unit-mthaddon-secrets-demo-1: 11:54:01 ERROR juju.worker.uniter.operation hook "config-changed" (via hook dispatching script: dispatch) failed: exit status 1

The code is available https://github.com/mthaddon/secrets-demo-operator and the charm is published on charmhub and can be deployed with juju deploy mthaddon-secrets-demo. You should hit the same error if you try scaling up the application.

7

looking into it. I think it must be something with ops, because secret-get from the unit appears to succeed

8

image It seems that the secret is there allright, but we can’t get the metadata and don’t know how to recover from that. I think with peer-scoped secrets there might be some subtlety we didn’t cater for yet in ops, we expect owner and consumer to be distinct entities but in this case the charm is both, however the follower unit does not have metadata access because it doesn’t have leadership.

Will look into this next week. @wallyworld thoughts on this?

1 Like
9

Only the secret owner can ask for metadata, ie for application owned secrets, this is the leader unit. All peer units can read the content of application secrets, but only leaders can access metadata.

10

Just wanted to note that we had a further discussion (Ian, Harry, Gustavo, Jon, me – I think it was) and decided to change the term “consumer” to “observer”. I’ve updated the secrets spec to match, and just updated this document with the new term.

I also made various clarifying updates, and updated the function names and code examples to match the API that’s now been merged (so it should be stable now). Thanks @ppasotti for the original doc!

1 Like