Pebble is secretly a FileWatcher?!

charm
,
1

In the event that a service cannot watch for config file changes and reload automatically, we have to implement restart logic in a charm ourselves. Traditionally this is done in code similar to this (pseudo) code:

if container.can_connect():
    try:
        running_config = yaml.safe_load(container.pull(CONFIG_PATH))
    except (FileNotFoundError, Error):
        ...

if running_config() != new_config:
    container.push(CONFIG_PATH, new_config)
    container.restart()
container.replan()

Alternatively, we can condense these 9 LOC to 2 LOC and 1 new LOC (_config_hash_for_auto_restart) in the Pebble Layer:

container.push(CONFIG_PATH, new_config)
container.replan()

layer = Layer(
    {
        "services": {
            "my-service": {
                "command": f"/usr/bin/my-service --config={new_config}",
                "startup": "enabled",
                "environment": {
                    "_config_hash_for_auto_restart": sha256(yaml.safe_dump(new_config)),
                },
            }
        },
    }
)

This new approach leverages the functionality of the Pebble replan function, which only restarts (and start startup-enabled) service(s) IFF the serviceā€™s Layer has changed. With our clever workaround, the hash of the new config file will differ from the previous hash Pebble has stored. The serviceā€™s Layer env var _config_hash_for_auto_restart forces workload container restarts conditionally and automatically!

This approach is inline with holistic charming i.e. reconciler pattern, also known as ā€œrebuild the worldā€.

Note: As of Python 3.7 a Dict keeping insertion order is a guaranteed language feature. Good thing we use Python >= 3.8 in charming!

6 Likes
2

a clever hack! I saw this in some code in parca at some point and I was rather confused, so I ended up stripping it. Maybe good to keep in mind that certain workloads support hot-reloading certain parts of their config, so we want to only hash the bits of the config that, if changed, require an actual restart.

But indeed this is rather a common situation; I wonder if it would make sense to add this functionality to pebble in some way.

I feel that the configuration files belong semantically in the layer definition. Suppose we extend a pebble layer to also include the config files (and only those that, if changed, require a restart); then the pebble layer semantics will already force a restart if that were to change.

@benhoyt WDYT?

3

I think thereā€™d be a pretty high bar for a ā€œrestart-if-changed: myconfig.yamlā€ kind of feature, but itā€™s not out of the question. I think this workaround is actually a fairly neat way to spell that, in lieu of such a feature.

4

uhm I see that, but what I had in mind is rather:

layer:
  name: foo
  command: mycommand
  environment: 
    - MYENV=1
  config: 
    - location: /etc/conf.conf
      content: | 
        some long configuration file

I think a nice feature of pebble is the declarative nature of the plan. Why is the config not declared at the same level too? Why if I pass a flag to my command it is ā€˜part of the planā€™, while if I want to pass it via config file it suddenly isnā€™t, and I have to interact with it imperatively?

1 Like