Event '<container>-pebble-ready'

doc
,
1

Event > List of events > Lifecycle events > <container>-pebble-ready

Source: ops.PebbleReadyEvent

The <container>-pebble-ready event is emitted once the sidecar container has started and a Unix socket is available for Pebble. There is one <container>-pebble-ready event for each container defined in charmcraft.yaml. This event allows the charm to configure how services should be launched.

This event is specific to Kubernetes sidecar charms and is only ever fired on Kubernetes deployments.

The pebble-ready event doesn’t guarantee the workload container is still up. For example, if you manually kubectl patch during (for example) install, then you may receive this event after the old workload is down but before the new one is up. For this reason it’s essential, even in pebble-ready event handlers, to catch ConnectionError when using Pebble to make container changes. There is a Container.can_connect() method, but note that this is a point-in-time check, so just because can_connect() returns True doesn’t mean it will still return True moments later. So, code defensively to avoid race conditions.

Moreover, as pod churn can occur at any moment, pebble-ready events can be received throughout any phase of a charm’s lifecycle. Each container could churn multiple times, and they all can do so independently from one another. In short, the charm should make no assumptions at all about the moment in time at which it may or may not receive pebble-ready events, or how often that can occur. The fact that the charm receives a pebble-ready event indicates that the container has just become ready (for the first time, or again, after pod churn), therefore you typically will need to reconfigure your workload from scratch every single time.

This feature of pebble-ready events make them especially suitable for a holistic handling pattern.

Observing this event in ops

When using the ops library, you can observe the event like you would any other:

self.framework.observe(
    self.on.<container>_pebble_ready, 
    self._on_pebble_ready,
)

The PebbleReadyEvent class is a subclass of WorkloadEvent, so it exposes a workload attribute which returns the ops.Container instance the event is associated with.

2
3

No permissions to edit

4

You should have them now!

5

We should probably mention explicitly that pebble-ready can be emitted before/after any of config-changed/start.

We should probably mention that wait_for_idle may return before the workload is ready, and a block_until the service, e.g. via HTTP API, returns a “ready” status, would probably be needed.

6

Made a note to follow up on this. Thanks, @sed-i! PS Feel free to intervene directly, if you wish.

7

So, that’s what I read about the pebble ready, but I went to do some testing around that. Take the upgrade-charm sequence:

stop(old charm) -> upgrade-charm(new charm) -> config-changed -> start -> <container>-pebble-ready

What I did is to add 1 minute sleeps to the upgrade-charm, config-changed and start handlers.

I kill the pod with kctl delete pod <pod name> -n <model name>. After the new container has been replaced and the new agent is in place, I connect to the pod with kctl exec . . . -c <workload> -n <model> -- /bin/bash

At this point, I can see that pebble is already running with a ps aux, and the charm code is still on the sleep for upgrade-charm. I would think that pebble, being ready well before config-changed or start, would trigger the event. But no, those events will trigger (with the sleeps) and only after that, <container>-pebble-event happens.

Is there something I’m missing here?

PS: I agree on the wait_for_idle comments. Very useful information :slight_smile: I’ll add that

8

Thanks @zmraul, I can confirm those observations on my end as well.

@jameinel, it seems that pebble-ready now always fires after start. Has anything changed in event sequencing logic or are we missing something?

9

pebble-ready fires once the other container has pebble actually up and running. It shouldn’t be guaranteed before/after start (and certainly could happen post start if the primary container restarts at any point)

10

Using sleep in install/config-changed/upgrade-charm, I was NOT able to make pebble-ready be emitted before start so far.

11

Juju has fairly implicit ordering of hooks, although not strictly defined, pebble-ready hooks fire just before an action or command could be run.

It is fairly guarenteed after install/upgrade, config-changed and start. There is some guarantee that pebble-ready will happen after relation-created and leadership hooks, only if those are already ready to run.

If there is a need for it, we could add another hook that fires as early as possible, e.g. pebble-early-ready, but we would need to understand your use case further to see when this should run, as pebble is likely to always be ready before the juju uniter has even downloaded the charm.

12

Thanks for the details @hpidcock!

I’m just trying to understand why even after I add long sleep in config-changed and upgrade-charm, I still see pebble-ready fire only after all of them.

13

Just to rephrase what Harry was saying. pebble-ready-event always fires after ‘start’. (Start is the point where a Unit agent starts to join its relations and can respond to things like juju run.) I had probably given bad information originally saying that it could fire at any time. Harry had actually done the work to not trigger pebble-ready before start (but it has been that way for all of 2.9).

14

Thanks for the clarifications @jameinel @hpidcock. I updated the “Emission sequence” section according to your input.

1 Like
15

I added a bunch of sleeps and when the charm is first deployed I consistently see pebble-ready after leader-elected but before config-changed and start.

16

Container.can_connect() is a point-in-time check, so try-except is technically still needed for the same reason (i.e. can-connect = True doesn’t mean that 1 millisecond after can-connect is still true).

I also think we should more clearly separate juju from ops in these kind of docs. I.e. I would try to avoid mentioning ops under the emission sequence section.

@ppasotti

17

both good points, I incorporated them in the doc. Thanks!

1 Like
18