Use dependency.Engine And catacomb.Catacomb

juju
1

Part of https://discourse.jujucharms.com/t/read-before-contributing/47, an opinionated guide by William Reade

See go doc ./worker/catacomb and go doc ./worker/dependency for
details; the TLDRs are roughly:

  • dependency.Engine allows you to run a network of
    interdependent tasks, each of which is represented by a
    dependency.Manifold which knows how to run the task, and what
    resources are needed to do so.
  • Tasks are started in essentially random order, and restarted when
    any of their dependencies either starts or stops; or when they
    themselves stop. This converges pretty quickly towards a stable set
    of workers running happily; and (sometimes) a few that are
    consistently failing, and all of whose dependents are dormant
    (waiting to be started with an available dependency).
  • Two tasks that need to share information in some way should
    generally not depend on one another: they should share a
    dependency on a resource that represents the channel of
    communication between the two. (The direction of information flow is
    independent of the direction of dependency flow, if you like.)
  • However, you can simplify workers that depend on mutable
    configuration, by making them a depend upon a resource that
    supplies that information to clients, but also watches for changes,
    and bounces itself when it sees a material difference from its
    initial state (thus triggering dependent bounces and automatic
    reconfiguration with the fresh value). See worker/lifeflag and
    worker/migrationflag for examples; and see worker/environ for
    the Tracker implementation (mentioned above) which takes advantage
    of environs.Environ being goroutine-safe to share a single value
    between clients and update it in the background, thus avoiding
    bounces.
  • You might want to run your own dependency.Engine, but you’re
    rather more likely to need to add a task to the Manifolds func in
    the relevant subpackages of cmd/jujud/agent (depending on what
    agent the task needs to run in).

…and:

  • catacomb.Catacomb allows you to robustly manage the lifetime of a
    worker.Worker and any number of additional non-shared Workers.
  • See the boilerplate in the worker.Worker section, or the docs, for
    how to invoke it.
  • To use it effectively, remember that it’s all about responsibility
    transfer. Add takes unconditional responsibility for a supplied
    worker: if the catacomb is Killed, so will be that worker; and if
    worker stops with an error, the catacomb will itself be Killed.
  • This means that worker can register private resources and forget
    about them, rather than having to worry about their lifetimes; and
    conversely it means that those resources need implement only the
    worker interface, and can avoid having to leak lifetime information
    via inappropriate channels (literally).

Between them, they seem to cover most of the tricky situations that come
up when considering responsibility transfer for workers; and since you
can represent just about any time-bounded resource as a worker, they
make for a generally useful system for robustly managing resources that
exist in memory, at least.

All Our Manifolds Are In The Wrong Place

…because they’re in worker packages, alongside the workers, and thus
severely pollute the context-independence of the workers, which can and
should stand alone.

The precise purpose of a manifold is to encapsulate a worker for use in
a specific context: one of the various agent dependency engines. It’s
at the manifold level that we define the input resources, and at the
manifold level that we (should) filter worker-specific errors and render
them in a form appropriate to the context.

(For example, some workers sometimes return dependency.ErrMissing or
dependency.ErrUninstall – this is, clearly, a leak of engine-specific
concerns into the wrong context. The worker should return, say,
local.ErrCannotRun: and the manifold’s filter should convert that
appropriately, because it’s only at that level that it makes sense to
specify the appropriate response. The worker really shouldn’t know it’s
running in a dependency.Engine at all.)

Next time someone has a moment while doing agent work, they should just
dump all the manifold implementations in appropriate subpackages of
./cmd/jujud/agent and see where that takes us. Will almost certainly
be progress…

Next: https://discourse.jujucharms.com/t/use-watchers-but-know-what-youre-doing/2745