When operating distributed applications, some actions should never happen everywhere at once. Restarting units, applying disruptive configuration changes, rotating credentials, or running maintenance tasks may be safe on one unit, but risky if several units execute them concurrently.
Juju gives charm authors excellent primitives for modelling applications, relations, leadership, hooks, and actions. However, Juju does not provide a native “rolling operation” mechanism: there is no built-in way to say, “run this task on each unit, but only one unit at a time, and retry safely if something goes wrong.”
The charmlibs.rollingops library fills that gap.
It provides a reusable coordination layer for charm authors who need to execute operations sequentially across units. Instead of each charm implementing its own locking, queueing, retry, rollingops exposes a common API that eases the task.
How charmlibs.rollingops work
The basic idea behind rolling operations is simple: units do not run disruptive operations immediately. Instead, they request a lock and only execute the operation once the lock is granted. This makes rolling operations asynchronous from the charm’s point of view. The charm requests the operation during a hook, but the actual work happens later, when the unit’s turn arrives.
When requesting the lock, the charm unit specifies a callback, optional arguments, and an optional maximum number of retries. Operations are added to a queue which is managed by the RollingOpsManager object living in the library.
A callback execution can have 3 outcomes:
-
RELEASE: The operation succeeded. Release the lock
-
RETRY_HOLD: The operation should be retried while keeping the lock on the same unit
-
RETRY_RELEASE: The operation should be retried later. The lock should be released so another unit can proceed
The callback specifies the desired outcome through its return value. The library handles retries and lock release automatically, and stops retrying when the configured retry limit is reached.
The library also performs deduplication: if the latest queued operation is identical to the new request, the new request is ignored. This prevents repeated hook executions from filling the queue with duplicate work.
Two Coordination Backends
The library currently supports two coordination modes:
The peer backend uses the charm’s peer relation to store operation queues and lock state. In this mode, the leader unit acts as the scheduler and grants the lock to one unit at a time.
This works well for coordinating rolling operations inside a single Juju application.
The etcd backend uses etcd as a distributed coordination backend. Units independently compete for the lock using etcd primitives, which allows rolling operations to be coordinated beyond a single application boundary.
This is useful when the operation must be serialized across a wider cluster, for example when several related applications must not perform disruptive work at the same time.
From the charm author’s point of view, enabling etcd-backed rolling ops is intentionally simple. The charm just integrates with an etcd application and the library handles the etcd client setup, distributed lock management, backend selection and retry logic.
requires
etcd:
interface: etcd_client
limit: 1
optional: true
When etcd is configured, charmlibs.rollingops prefers it as the primary backend. The peer relation is still used as a durable charm-side state, so operations can be mirrored and recovered if etcd becomes unavailable. In other words: etcd gives stronger distributed coordination, while the peer backend provides a local fallback path.
Synchronous locks
The main execution model in charmlibs.rollingops is the asynchronous lock. The library also provides a synchronous lock mechanism for cases where the charm cannot defer the work. This is mainly intended for teardown or finalization paths, where a charm may need to serialize a critical operation while the current hook is still running.
Synchronous locks should be used carefully, because the hook remains blocked while waiting for the lock.
When the etcd backend is configured and available, synchronous locking is handled natively by the library: it acquires and releases the etcd lock internally.
When etcd is not integrated, the peer backend alone is not enough to provide synchronous locking. Peer relations may not be available or reliable during teardown. In that case, the charm author must provide a custom implementation of SyncLockBackend, a small class that implements the acquire/release logic for the charm’s own locking mechanism.
The custom backend is also used as a fallback when etcd is configured but unavailable. This lets charms use etcd for synchronous locking when possible, while still having a charm-specific fallback path when the library needs to fall back to peer-based behavior.
A Single Charm-facing API
Charm code does not need to implement separate rolling-operation flows for each backend. The charm creates a RollingOpsManager, registers callbacks, and requests locks through the same API. You can import the library and use it as follows:
from charmlibs.rollingops import RollingOpsManager, OperationResult
sync_stop_backend = SyncLockStopBackend()
self.restart_manager = RollingOpsManager(
charm=self,
peer_relation_name="restart",
etcd_relation_name="etcd",
cluster_id="cluster-12345",
callback_targets={
"restart": self._restart,
},
sync_lock_targets={
"stop" : sync_stop_backend,
},
)
You can request a lock on a hook execution:
def _on_config_changed(self, event):
self.rollingops.request_async_lock("restart", kwargs={'force': True}, max_retry=2)
The callback is regular charm code:
def _restart(self, force: bool = False):
if not self.unit.is_ready():
return Operation.RETRY_RELEASE
self.unit.status = MaintenanceStatus("Restarting")
restart_service(force)
self.unit.status = ActiveStatus()
return OperationResult.RELEASE
For synchronous cases:
with self.restart_manager.acquire_sync_lock(
backend_id="stop",
timeout=60,
):
self._stop_service()
Choosing peer vs etcd
Use the peer backend when the operation only needs to be serialized within one Juju application.
Peer-backed rolling ops are a good fit for:
-
Rolling restarts inside one application
-
Sequential config changes across units of the same charm
-
Deployments where adding etcd would be unnecessary operational overhead
-
Cases where leader-driven scheduling is enough
Use the etcd backend when rolling operations need cluster-level coordination.
Etcd-backed rolling ops are a good fit for:
-
Coordinating disruptive operations across multiple applications
-
Avoiding concurrent maintenance across a larger system
-
Charms that already depend on etcd or can reasonably integrate with it
-
Cases where leader-based scheduling inside one app is not enough
Try it in your charm
The charmlibs.rollingops has already been adopted by Charmed MongoDB.
If your charm has operations that should run one unit at a time, charmlibs.rollingops is ready to help. You can find the package in the charmlibs repository and the documentation here.
Contributions, feedback, and real charm use cases are very welcome.
