Securely Pass Secrets to Services with `systemd-creds`

charm
, ,
1

I set out on a bit of an adventure recently, trying to figure out a way to securely pass secrets to services in machine charms. What follows is my “blogisized” discovery.

I’m sure this can still use a lot of improvement. It was a long journey to get here due to several modifications needed and some bugs that made debugging a little tricky. But, I think it’s headed in the right direction.

Sometimes, you need to provide a secret to a process, especially a daemon. In the world of Juju charms, this often means sensitive data like API keys, database passwords, or private keys. The goal is to avoid hardcoding them or exposing them in logs, ensuring that the secret’s availability is as constrained as possible. This is particularly crucial when working on something like the Vault charm, which is a secrets store itself and requires the highest level of security.

Common Approaches and Their Shortcomings

Configuration Files

A simple, yet insecure, method is to store secrets in a configuration file.

The data flow looks something like this:

This method has significant disadvantages:

  1. Unencrypted at rest: The secret is stored on disk in plain text.
  2. Increased attack surface: If the service is compromised and can read from the disk, the unencrypted secret is exposed.

Environment Variables

Using environment variables is a common approach that addresses some of the issues with config files.

The data flow is slightly different:

While environment variables are less likely to end up in version control, they come with their own set of drawbacks:

  • Inheritance: They are inherited by child processes.
  • Leakage: They can be accessed by other processes on the same machine (e.g., via ps or /proc/PID/environ) and can be leaked through logs.
  • Security: They are not encrypted at rest and lack auditing capabilities, as detailed in this Node.js security blog post.
  • Snap-specific issue: It can be challenging to pass an environment variable to a service managed by snapd, as it won’t be forwarded automatically.
  • Binary data: Environment variables can have issues with binary data, as they are meant for storing strings. There is a maximum size limit which applies to all environment variables combined.

Snap Configs

An older proposal for snaps was to use their configuration system. However, this approach, which stores secrets in a plain text file at /var/lib/snapd/state.json, suffers from the same vulnerabilities as the configuration file method.

The Solution: Systemd Credentials

systemd-creds offers a modern, secure solution for providing secrets to services. This system leverages systemd’s built-in credential management to handle secrets with a high degree of security.

Read more about it on the official Systemd Credentials page.

In a nutshell, Systemd Credentials provides several key advantages:

  • Temporary and restricted: Credentials are made available only when the service is active and are released upon deactivation. Access is restricted to the service’s user and is not propagated down the process tree.
  • Encrypted: Secrets are encrypted at rest, either using a key from a TPM2 chip or one stored in /var/. This process is designed to be automatic, so if you have a TPM2 chip you will automatically benefit from it.
  • Memory protection: Credentials are placed in non-swappable memory, preventing them from being written to the disk.

The secure data flow now looks like this:

Implementation

To implement this in a Juju charm, there are three main steps:

Encrypt the Secret

We can use systemd-creds to encrypt the secret. We’ll write the secret to /etc/credstore.encrypted/ where systemd will automatically search for it, making it easy to load later.

subprocess.run(
    [
        "systemd-creds",
        "encrypt",
        f"--name=my_secret",  # Credential name
        "-",  # Take input from stdin (instead of a file)
        f"/etc/credstore.encrypted/my_secret",  # Encrypted credential output location
    ],
    input=secret,
    text=True,
    check=True,
)

Generate a Systemd Drop-In File

A drop in file is effectively an overlay in systemd. It allows you to add or override settings for a service without modifying the original service file.

Snap creates service files for each service contained in the snap. These files are located in /etc/systemd/system/snap.<snap-name>.<service-name>.service. To modify the service, you create a drop-in file in /etc/systemd/system/snap.<snap-name>.<service-name>.service.d/. So, in our Vault example, we add a service file to /etc/systemd/system/snap.vault.vaultd.service.d.

[Service]
LoadCredentialEncrypted=my_secret

After creating or modifying a drop-in file, you need to reload the systemd daemon.

subprocess.run(["systemctl", "daemon-reload"], check=True)

You also need to restart the service to pick up the new configuration. This can be done with a snap restart call.

Read the Credential in Application Code

The secret should now be available to the service in a file located at ${CREDENTIALS_DIRECTORY}/my_secret. The file is owned by the user running the service and has strict access permissions. Furthermore, it is stored in non-swappable memory and is automatically removed when the service stops. The only place this secret exists on disk is in encrypted form.

Unfortunately, in our Vault example, Vault cannot read the token from a file until version 1.19, so we’re stuck injecting this token as an environment variable for now. We use a script to read the secret from the file and export it as an environment variable before starting Vault.

Notes