When to set non-active juju status

charm
,
1

Often, an admin looks at the output of juju status and sees a green active/idle, even though something is definitely wrong - for example, no space left on device.

With separation of concerns in mind, we should ideally use juju status (and juju log) for operator concerns, and standard observability for workload concerns. This separation can be a bit fuzzy when the charm itself is instrumented, since the charm’s own logs and traces may also be forwarded to the same observability backend. That is a good thing - it enables alerts and dependable support. Technically, “everything” can and should end up in your observability backend.

But which situations belong in juju status? Pure operator concerns such as snap management are obvious, but some situations are both a workload and an operator concern, and would benefit from being reflected in juju status in addition to the observability backend.

image
image1600×900 99.9 KB
(Edit a copy of this diagram)

Situation Workload concern Operator concern
no space left on device back-pressure; service outage charm config options that affect disk usage; know-how on safe purging (destructive data loss), but must not be automated - must be triggered by admin
x509 error back-pressure; service outage relation data incomplete, incorrect or outdated; charm depends on root ca certs the oci image didn’t include

In other words, the “concern” is not necessarily “juju status” vs. “observability”, but rather more contextual and fluid - e.g. “handover vs operation”, “admin vs SRE”, “ssh vs gui”, “rapid response vs in-depth investigation”. There is also no exclusivity restriction: we can log and block for the same reason. Looking at juju status is similar to running htop or ps aux | grep ... - it’s familiar, easy and straightforward, and currently some admins rely on it.

Why set status in addition to having an alert rule

A couple of scenarios where we should set status (e.g. blocked) in addition to having an alert rule:

  1. Reflecting an operational issue with an alert rule may be too challenging in a given context. For example, to get “no space left on device” alerts out-of-the-box for a Persistent Volume in K8s, we’d need to use charmed k8s, ensure the team included an alert for this, and that our substrate is using a new enough revision of the k8s charm that includes observability integrations.
  2. Network partition or complete outage of the observability cluster. If everything is active, the admin’s task is much more difficult.
  3. Users getting started with Juju who do not yet understand what’s going on. It doesn’t make sense to ask them to deploy an observability model just to get a feel for charmed operators.
  4. Minimal or edge deployments that run without observability.

image
image1214×551 50.6 KB
(Edit a copy of this diagram)

Where do we draw the line?

It would be useful for admins to see a non-active status in any the following situations:

  • Anything that currently causes an outage (regardless of whether an admin can do anything about it) that the charm can determine without poking the workload’s API. For example:
    • Incomplete integrations graph, e.g. mandatory relation pairs are not fulfilled.
  • Anything that currently causes a partial outage that the charm can determine without poking the workload’s API, and an admin is needed to resolve.
  • Model departure, i.e. there is no outage but the workload is operating in a context different from what the juju model strictly prescribes. For example:
    • Validation of forwarded alert rules failed.

How non-active status messages help with observability

The Juju team is working on improving the instrumentation of the controller. When “juju status” is mirrored in the observability backend, we will be able to have dashboards and alerts for non-active statuses. This could be reflected as metrics or log lines collected from juju show-status-log. Rendering valuable non-active status messages in the charm today, would improve the quality of diagnosis from within the observability platform in the future.

Summary

Admins today and SREs in the future would benefit if certain issues were reflected using non-active juju status messages. Messages that are worthwhile surfacing as a juju status usually have something to do with an outage or model departure.

References

2 Likes
3

Amazing description of the topic and very very nice examples @sed-i!

In simple terms, I’d also make the case that the workload status in juju status should be the charm’s most informed understanding of workload health. In most discussions we’ve had regarding the use of juju status for the purpose of surfacing workload-level and workload+charm-level health status, a counterargument has roughly been:

To what extent do you surface issues through juju status? If you set the workload status to blocked due to disk pressure, should you do the same for “CPU use too high”? What about for “memory use too high”?

This is a valid concern, particularly because it would be unrealistic to expect the charm to continuously probe for a list of problematic issues such as disk pressure and surface them through juju status in a way that is “unnoisy” and actionable. However, as I think you’re hinting at (and I share the same thinking), the idea is to use the workload health entry under juju status to surface workload-level concerns that are caused by charm-level issues.

To add to the list of examples you provided for cases where setting a non-active workload status is warranted, we can mention:

  • Scenarios in which a machine charm fails to start the snap it’s operating. This, at least in my opinion, has been somewhat of a recurring issue across a wide range of machine charms. You deploy the charm, relate to x other charms which send some data over relation data to be used for writing the workload config. Assume the charm executes a relation-changed event, takes the relation data, writes the workload config, and attempts to restart the snap. If the snap fails to start, there is a significant chance that the config that the charm wrote based on what came over relation data is invalid. Regardless, the fact that the snap has not restarted successfully means the workload is NOT running. It then only makes sense to indicate this by setting the “workload status” to something other than active.

In the example above, the non-active status is indicating that there is a workload-level issue that is most likely caused in the charm-level layer.

4

IMO one more example that would warrant a non-active status, though likely blocked, is the case when more charms are needed to form a working product. If we deploy tempo-coordinator, but it isn’t related to at least one tempo-worker and a charm / integrator that provides access to an s3-like backend, it isn’t able to store traces yet, so this is a good case for blocked.

5

Thanks @sinap!

6

Agreed. If we change up the example a bit: in a given deployment, if we take away a mandatory relation then it’s an outage. So I think this would fall under the anything that currently causes an outage criterion. Wdyt?

7

Yes, I agree. I just think it’s worth pointing out too within that scope.

8

Good point, updated, thanks!

10

The general guidance is that you should go into blocked if you need a human to get you into working order. Out-of-disk space is certainly one of those cases. It is a bit of a shame, though, as it is a concern for all charms, so it would be nice if Juju considered that one generically. The only specific issue is which disk, how much space, etc. Is it blocked if any storage is low, is it blocked only if a specific one is low?

11

Good point. I think if pebble had presets for certain checks, it could be handy.

For example, goss has many kinds of built-in check types. Drawing from that approach, if pebble added some sort of kind to the pebble check spec, it could make things easier. Since pebble is only on k8s now, maybe we could have some abstraction for “checks” that could render pebble checks and systemd unit files in the spirit of goss.