An issue I’ve seen a few times (here’s a post from @sed-i for example) is a charm wanting to be woken up to do work when the application (workload) is ready, rather than when Pebble or Juju is ready.
If you configure Pebble checks, you do get a PebbleCheckRecovered event in Juju 3.6. However, it’s a race as to whether you’ll get this when the application is first ready: if it’s slow enough to be ready that the checks start failing then you will; if it’s fast enough that the checks don’t fail (often enough to hit the threshold) then you won’t, because it’s a recovered event, not a passed event.
@ppasotti has built a “tempo ready” event using Pebble custom notices (I think you could do this with a check as well). The key bits are roughly this handler:
def _on_workload_pebble_custom_notice(self, event: PebbleNoticeEvent):
if event.notice.key == self.workload_ready_notice_key:
self.workload_container.stop("workload-ready")
# more things to do here based on the workload being ready
and roughly this layer, which hits a /ready endpoint to determine if the workload is ready:
return Layer(
{
"services": {
"workload-ready": {
"override": "replace",
"summary": "Notify charm when workload is ready",
"command": f"""watch -n 5 '[ $(wget -q -O- http://localhost:{self.workload_http_server_port}/ready) = "ready" ] &&
( /charm/bin/pebble notify {self.workload_ready_notice_key} ) ||
( echo "workload not ready" )'""",
"startup": "disabled",
}
},
}
)
If the application has the ability to configure a hook when ready (like an nginx init script) then you can obviously do this more simply by just adding a pebble notify there. However, there are a lot of applications that don’t offer this.
Question 1: does your charm already have some sort of “application ready” event, and if so, how are you implementing that? Links to examples would be really great; even if the answer is “we can already do this” then a how-to guide that includes links to a bunch of examples would be useful.
Question 2: if you don’t have an “application ready” event, is this something you wished you had? (K8s only? Machine only? Machine and K8s?)
Question 3: do your applications have a way to distinguish between “running” and “ready”? It would be great to have some examples of what this is - are you looking for a specific log message, does answering on a port mean “ready”, is there a /ready endpoint, etc. Pebble/Juju/ops isn’t going to be able to solve this if you don’t, but it would be helpful to know what sort of thing you’d trigger on.
Somewhat related to this, Pebble checks are currently either “up” or “down”, even if they have not run for the first time yet. If they have a level then for K8s we need to transform whatever the statis is into a binary answer, but question 4 would you find checks more useful if there was a different initial state, like “pending” or “unknown”?