Juju Actions: opt-in to new behaviour from Juju 2.8

[wallyworld (Ian Booth)]

The UX around Juju Actions has undergone a fairly radical re-design to address some long standing issues and deliver some nice usability improvements.

An early version of this was delivered in Juju 2.7. Coming in Juju 2.8 is a more polished iteration.

Key points:

• terminology changes
• running an action defaults to synchronous (block until done)
• running actions can be cancelled
• new ability to display progress log messages
• cleaner plain text output option
• stdout and stderr included in output
• numeric ids instead of UUIDs
• ability to list pending, running, completed actions
• ability to watch progress of a running action

New commands

juju run <action>
juju operations [filter options]
juju show-operation <ID>
juju show-task <ID>
juju cancel-task <ID>

Getting Started

To use the new UX, the only thing you’ll need to do is enable the actions-v2 feature on the client. The backend controller infrastructure is compatible with both the old and new UX.

export JUJU_FEATURES=actions-v2

Terminology changes

The terminology around this feature changes as follows:

• enqueued actions on individual units are called tasks
• running an action run across one or more units is an operation
• an operation comprises 1 or more tasks
• juju run-action becomes juju run
• juju run becomes juju exec
• juju show-action-output becomes juju show-operation / juju show-task

Charm developer changes

As a charm developer, there’s a new hook command (and corresponding charm helpers API) to log action progress messages.

$ action-log --help
Usage: action-log <message>

Summary:
record a progress message for the current action

Package charmhelpers.core

def action_log(message):
    """Write an action progress message"""

Feature walk through

We’ll deploy a charm with a sample action hacked together to try out some of the new features. It happens to be a k8s charm, to help show that actions now also work on k8s.

$ cat actions/hello
#!/bin/bash
action-set result-map.message="Hello $(action-get who)!"
action-set outcome="maybe"
echo Goodbye cruel world
echo This message goes to stderr >&2
echo Foo bar
for i in {1..5}
do
   action-log "Welcome $i times"
   sleep 1
done
exit 3

$ juju deploy mycharms/mariadb-k8s -n 2
$ juju actions mariadb-k8s
Function  Description
hello     Hello World.

$ juju show-action mariadb-k8s hello
Hello World.

Arguments
who:
  type: string
  description: The person to say hello to.

Running actions

We’ll run the action with default behaviour - synchronous and plain text output.

$ juju run mariadb-k8s/0 hello who=world
Running operation 1 with 1 task
  - task 2 on mariadb-k8s/0

09:39:43 Welcome 1 times
09:39:44 Welcome 2 times
09:39:45 Welcome 3 times
09:39:46 Welcome 4 times
09:39:47 Welcome 5 times

info:
  host: mariadb-k8s-0
  message: Hello world!
  outcome: maybe

Goodbye cruel world
Foo bar
This message goes to stderr

Note that the plain text output omits tasks metadata like enqueue time etc and just shows progress logs, structured output data, and stdout/stderr.

YAML output shows everything. We can also choose to display timestamps in UTC.

$ juju run mariadb-k8s/0 hello who=world --format yaml --utc
Running operation 3 with 1 task
  - task 4 on mariadb-k8s/0

Waiting for task 4...
03:20:48 hello sailor
03:20:48 Welcome 1 times
03:20:49 Welcome 2 times
03:20:50 Welcome 3 times
03:20:51 Welcome 4 times
03:20:52 Welcome 5 times

mariadb-k8s/0:
  id: "4"
  log:
  - 2020-02-25 03:20:48 +0000 UTC hello sailor
  - 2020-02-25 03:20:48 +0000 UTC Welcome 1 times
  - 2020-02-25 03:20:49 +0000 UTC Welcome 2 times
  - 2020-02-25 03:20:50 +0000 UTC Welcome 3 times
  - 2020-02-25 03:20:51 +0000 UTC Welcome 4 times
  - 2020-02-25 03:20:52 +0000 UTC Welcome 5 times
  message: command terminated with exit code 3
  results:
    info:
      host: mariadb-k8s-0
      message: Hello world!
    return-code: 3
    stderr: |
      This message goes to stderr
    stdout: |
      Goodbye cruel world
      Foo bar
  status: failed
  timing:
    completed: 2020-02-25 03:20:59 +0000 UTC
    enqueued: 2020-02-25 03:20:45 +0000 UTC
    started: 2020-02-25 03:20:48 +0000 UTC
  unit: mariadb-k8s/0

To see a particular operation or task (whether completed or not), use the show-operation or show-task command. The command will return immediately with the current state of the operation or task, unless --watch is used which will block and report the progress of a running operation or task until it completes. The --wait X option is also supported, to allow an upper bound on how long to wait for the operation or task to complete.

show-operation produces a detailed YAML (or JSON) record of the operation and all associated actions.

show-task by default produces a plain text output that is quite concise, and omits any progress log messages. Use YAML (or JSON) to see full output. As with other related commands, --utc can be used if desired.

Note: we’ve been running an action on one unit - the distinction between operations and tasks becomes more apparent when an action is run across multiple units.

$ juju run mariadb-k8s/0 mariadb-k8s/1 hello who=world
Running operation 5 with 2 tasks
  - task 6 on mariadb-k8s/0
  - task 7 on mariadb-k8s/1

Waiting for task 6...
Waiting for task 7...
mariadb-k8s/0:
  id: "6"
  output: |
    info:
      host: mariadb-k8s-0
      message: Hello world!
mariadb-k8s/1:
  id: "7"
  output: |
    info:
      host: mariadb-k8s-1
      message: Hello world!

$ juju show-operation 5
summary: hello run on unit-mariadb-k8s-0,unit-mariadb-k8s-1
status: failed
action:
  name: hello
  parameters:
    who: world
timing:
  enqueued: 2020-02-25 13:25:29 +1000 AEST
  started: 2020-02-25 13:25:33 +1000 AEST
  completed: 2020-02-25 13:25:57 +1000 AEST
tasks:
  "6":
    host: mariadb-k8s/0
    status: failed
    timing:
      enqueued: 2020-02-25 13:25:29 +1000 AEST
      started: 2020-02-25 13:25:46 +1000 AEST
      completed: 2020-02-25 13:25:57 +1000 AEST
    log:
    - 2020-02-25 13:25:46 +1000 AEST hello sailor
    - 2020-02-25 13:25:46 +1000 AEST Welcome 1 times
    - 2020-02-25 13:25:47 +1000 AEST Welcome 2 times
    - 2020-02-25 13:25:48 +1000 AEST Welcome 3 times
    - 2020-02-25 13:25:49 +1000 AEST Welcome 4 times
    - 2020-02-25 13:25:50 +1000 AEST Welcome 5 times
    message: command terminated with exit code 3
    results:
      info:
        host: mariadb-k8s-0
        message: Hello world!
      return-code: 3
      stderr: |
        This message goes to stderr
      stdout: |
        Goodbye cruel world
        Foo bar
  "7":
    host: mariadb-k8s/1
    status: failed
    timing:
      enqueued: 2020-02-25 13:25:29 +1000 AEST
      started: 2020-02-25 13:25:33 +1000 AEST
      completed: 2020-02-25 13:25:45 +1000 AEST
    log:
    - 2020-02-25 13:25:33 +1000 AEST hello sailor
    - 2020-02-25 13:25:33 +1000 AEST Welcome 1 times
    - 2020-02-25 13:25:35 +1000 AEST Welcome 2 times
    - 2020-02-25 13:25:36 +1000 AEST Welcome 3 times
    - 2020-02-25 13:25:37 +1000 AEST Welcome 4 times
    - 2020-02-25 13:25:38 +1000 AEST Welcome 5 times
    message: command terminated with exit code 3
    results:
      info:
        host: mariadb-k8s-1
        message: Hello world!
      return-code: 3
      stderr: |
        This message goes to stderr
      stdout: |
        Goodbye cruel world
        Foo bar

$ juju show-task 6
info:
  host: mariadb-k8s-0
  message: Hello world!

Goodbye cruel world
Foo bar
This message goes to stderr

The YAML output again shows everything, including log messages.

$ juju show-task 6 --format yaml
id: "6"
log:
- 2020-02-25 13:25:46 +1000 AEST hello sailor
- 2020-02-25 13:25:46 +1000 AEST Welcome 1 times
- 2020-02-25 13:25:47 +1000 AEST Welcome 2 times
- 2020-02-25 13:25:48 +1000 AEST Welcome 3 times
- 2020-02-25 13:25:49 +1000 AEST Welcome 4 times
- 2020-02-25 13:25:50 +1000 AEST Welcome 5 times
message: command terminated with exit code 3
results:
  info:
    host: mariadb-k8s-0
    message: Hello world!
  return-code: 3
  stderr: |
    This message goes to stderr
  stdout: |
    Goodbye cruel world
    Foo bar
status: failed
timing:
  completed: 2020-02-25 13:25:57 +1000 AEST
  enqueued: 2020-02-25 13:25:29 +1000 AEST
  started: 2020-02-25 13:25:46 +1000 AEST
unit: mariadb-k8s/0

You can run an action and have it run in the background.

$juju run mariadb-k8s/0 hello who=world --background

Scheduled operation 8 with task 9
Check operation status with 'juju show-operation 8'
Check task status with 'juju show-task 9

To see progress of a running operation or task:

$ juju show-task 9 --watch
13:32:40 hello sailor
13:32:40 Welcome 1 times
13:32:41 Welcome 2 times
13:32:42 Welcome 3 times
13:32:43 Welcome 4 times
13:32:44 Welcome 5 times

info:
  host: mariadb-k8s-0
  message: Hello world!

Goodbye cruel world
Foo bar
This message goes to stderr

juju show-operation 8 --watch would also work here.

To cancel a running task:

juju cancel-task 9

Cancelled tasks when viewed have a status of Aborting while waiting to be stopped, and then Aborted.

Listing operations

Actions which have been run are enqueued as a task on the unit; all tasks queued via a given run command are an operation. Enqueued tasks stay pending until they are executed, and then complete (successfully or not). The new operations (list-operations) command can be used to see what operations exist, optionally filtered by:

• unit
• application
• action name
• status (pending, running, completed, failed, aborting, aborted)

show-operation or show-task can then be used to inspect a particular operation in more detail.

When both unit(s) and application(s) are specified, operations for any of the individual units and any unit of the applications are included.

$ juju operations --status=running,completed,failed --actions=hello --apps=mariadb-k8s
Id  Status  Started              Finished             Task IDs  Summary
 1  failed  2020-02-25T13:17:03  2020-02-25T13:17:19  2         hello run on unit-mariadb-k8s-0
 3  failed  2020-02-25T13:20:48  2020-02-25T13:20:59  4         hello run on unit-mariadb-k8s-0
 5  failed  2020-02-25T13:25:33  2020-02-25T13:25:57  6,7       hello run on unit-mariadb-k8s-0,unit-mariadb-k8s-1
 8  failed  2020-02-25T13:31:45  2020-02-25T13:31:56  9         hello run on unit-mariadb-k8s-0 

Use YAML to see more information about each task. --utc works as well.

$ juju operations --status=running,completed,failed --actions=hello --apps=mariadb-k8s --format yaml --utc
"1":
  summary: hello run on unit-mariadb-k8s-0
  status: failed
  action:
    name: hello
    parameters:
      who: world
  timing:
    enqueued: 2020-02-25 03:16:59 +0000 UTC
    started: 2020-02-25 03:17:03 +0000 UTC
    completed: 2020-02-25 03:17:19 +0000 UTC
  tasks:
    "2":
      host: mariadb-k8s/0
      status: failed
      timing:
        enqueued: 2020-02-25 03:16:59 +0000 UTC
        started: 2020-02-25 03:17:03 +0000 UTC
        completed: 2020-02-25 03:17:19 +0000 UTC
      message: command terminated with exit code 3
"3":
  summary: hello run on unit-mariadb-k8s-0
  status: failed
  action:
    name: hello
    parameters:
      who: world
  timing:
    enqueued: 2020-02-25 03:20:45 +0000 UTC
    started: 2020-02-25 03:20:48 +0000 UTC
    completed: 2020-02-25 03:20:59 +0000 UTC
  tasks:
    "4":
      host: mariadb-k8s/0
      status: failed
      timing:
        enqueued: 2020-02-25 03:20:45 +0000 UTC
        started: 2020-02-25 03:20:48 +0000 UTC
        completed: 2020-02-25 03:20:59 +0000 UTC
      message: command terminated with exit code 3
...

[ec0 (James Hebden)]

This is awesome Ian - I’ve been using the juju-v3 flag for a little while now and love the changes.

As additional feedback, and if part of this work unblocks being able to pass files back from the units via the controller as the result of actions, that would improve UX as well IMO. I have often needed to have a function return a file back to the CLI (such as backing up data or certificates stored in the unit).

[wallyworld (Ian Booth)]

Glad you like the new UX. We have plans next cycle to introduce the concept of sub-tasks. The top level function invocation is an operation and operations can spawn tasks which are grouped under the top level operation. Details are still in flux. Note that the next 2.7 rc5 will use list-operations and show-operation.

We also will be introducing the ability to cancel running operations.

Unfortunately, there’s no immediate plans to be able to stream files resulting from running a function. This is something we definitely need to (and want to) do but for next cycle our todo list is somewhat overflowing. If there’s time, we can try to squeeze it in but can’t make any promises.

[wallyworld (Ian Booth)]

Bumping post so folks can see what’s new.

[chris.sanders]

Ian,
When the sample output above shows “Waiting for task 6…” are those running parallel or serial? It would be really nice if actions and the ‘exec’ command would run in parallel it makes a big difference with multiple units involved.

[erik-lonroth (Erik Lönroth)]

@chris.sanders

it makes a big difference with multiple units involved.

Agree alot.

I have potentially thousands of units in SLURM. A means to manage that is critical for operating large models.

[wallyworld (Ian Booth)]

Actions are run in parallel across machines, but serially on each machine. When you run an action, you can specify multiple target units for that action.

If a machine hosts several units, and an action has been queued up on multiple units, they will execute one at a time on that machine, as each action invocation grabs a global machine lock.

[pmatulis (Peter Matulis)]

@wallyworld

• juju run-action becomes juju run
• juju run becomes juju exec
• juju show-action-output becomes juju show-operation / juju show-task

Do we get backwards compatibility for a while or is this a hard change?

[wallyworld (Ian Booth)]

The new UX is behind a feature flag to give people time to transition.
Because both schemes have a juju run command, it’s not possible to have both at the same time.

[timClicks (Tim McNamara)]

So what will be happening in Juju 2.8? Which set of commands is going to be the default?

[wallyworld (Ian Booth)]

The new UX is behind a feature flag

[timClicks (Tim McNamara)]

Oh, okay. So this isn’t really coming to 2.8 for the vast majority of users. I’ll need to rework our upcoming documentation.

[wallyworld (Ian Booth)]

It’s a client only feature flag - trivial to turn on. I would encourage everyone to use the new UX. We just can’t change the defaults in a point release.
So it’s opt in but easy to do and we should document/encourage that.

[timClicks (Tim McNamara)]

@wallyworld

client only feature flag - trivial to turn on.

Spoken as a core developer

If you look at the docs for feature flags, there are massive warning lights saying that they shouldn’t be used for production systems.

I’ve just added a suggestion that might make enabling feature flags more accessible:

Change the feature flag environment variable to JUJU_FEATURES

An idea, let’s simplify the feature flag that people need to use to enable feature flags. The current one is unwieldy and all of the documentation surrounding them notes that they are experimental, e.g.: JUJU_DEV_FEATURE_FLAGS This setting takes effect on an environment only at bootstrap time. Unstable or pre-release features are enabled only when the feature flag is enabled prior to bootstrapping the environment. JUJU_DEV_FEATURE_FLAGS=<flag1,flag2> juju bootstrap Unforeseen and detrimental…

[wallyworld (Ian Booth)]

There’s a difference between server side feature flags (yes, do not use in prod yada yada) and ones only used to tweak client / CLI options.

[bloodearnest (Simon Davy)]

Hi all.

Some thoughts mostly from the view point of scripting/automating actions as part of a pipeline:

a) Very nice to see yaml dicts indexed by user friendly ‘service/n’ keys, rather than internal ‘unit-service-n’ keys, makes life much easier thank you.

b) One issue we’ve had with current actions is difference in yaml format between juju run-action --wait --format yaml and juju show-action-output <id> --format yaml (the latter has an extra dict around it). It seems like this fixes that (in that the yaml output is always in a dict keyed by unit), great!

Some questions:

1. Does juju run <units> <action> exit with non-zero return code if the action fails on any unit? For me, in an ideal world, in the single unit action case, it would exit with the same exit code that the action exited with.

2. Is seems the output on juju run --format yaml is mixed between client logging, action output and yaml. This would make it very difficult to consume from a script. If I’ve specified --format=yaml, I would expect to be able to feed the output to a yaml parser, but from that examples, it looks like this would break?

Both the above are related to the same use case. I have a deployment pipeline that automates juju orchestration. I want to run an action (e.g. db migration) as part of this, and I want everything to come to halt if this fails. Right now, doing this is awkward, before juju run-action --wait we had to use show-action-output and manually parse the yaml for success/failure. We recently switched to use --wait, but we still need to parse the yaml (which is a different structure than show-action-output). 1) would make this trivial for us, for example.

Also, a more off the beaten path question:

We automate many self authored similar looking charms with the same pipeline. Some of these services have databases, we include a migrate action on those charms, but not others. Because of this, our general purpose deployment pipeline tooling has to check if a migrate action exists before trying to run it. While this is reasonable, it is cumbersome, as involves yet more subprocesses and yaml parsing.

Would a flag like --ignore-missing or --if-exists or similar be an option? Would exit with 0 if the action is not there? I realise this is a bit of a niche case, but again, it’s coming from the automation perspective rather than the manual operation perspective.

[wallyworld (Ian Booth)]

Thanks for trying out the new UX and for the great feedback.

To answer the questions…

1. juju run <units> <action> returns a 0 error code if the actions are successfully queued to be executed. The YAML result returns the error code for the invocation on each unit.

2. Logging / user messages go to stderr, the YAML goes to stdout. So if you just want the YAML to feed to a parser, you can do something like:

$ juju run --format yaml ubuntu-lite/0 ubuntu-lite/1 hello who=world 2> /dev/null
ubuntu-lite/0:
  id: "11"
  log:
  - 2020-07-03 09:50:53 +1000 AEST hello sailor
  results:
    result-map:
      message: Hello world!
      time-completed: Thu Jul  2 23:50:53 UTC 2020
    return-code: 0
    stderr: |
      This message goes to stderr
    stdout: |
      Goodbye cruel world
  status: completed
  timing:
    completed: 2020-07-03 09:50:53 +1000 AEST
    enqueued: 2020-07-03 09:50:51 +1000 AEST
    started: 2020-07-03 09:50:53 +1000 AEST
  unit: ubuntu-lite/0
ubuntu-lite/1:
  id: "12"
  log:
  - 2020-07-03 09:50:53 +1000 AEST hello sailor
  message: exit status 3
  results:
    result-map:
      message: Hello world!
      time-completed: Thu Jul  2 23:50:53 UTC 2020
    return-code: 3
    stderr: |
      This message goes to stderr
    stdout: |
      Goodbye cruel world
  status: failed
  timing:
    completed: 2020-07-03 09:50:53 +1000 AEST
    enqueued: 2020-07-03 09:50:51 +1000 AEST
    started: 2020-07-03 09:50:53 +1000 AEST
  unit: ubuntu-lite/1

Note the --format yaml option includes everything, including stdout, stderr, return-code. You can still get a condensed version of the YAML by leaving off the --format option.

After the action has run, yoy can also see the result YAML using show-task [--format yaml].

In terms of the question about having to parse the YAML to get the return codes for each action on each unit, it doesn’t do that because the run action creates a top level operation which is queued. Each sub-task of that operation is an invocation of the action on a unit, and these are fired off asynchronously. In non-background mode, the CLI does not exit but watches for the tasks to complete. It can be interrupted at any time and then you can use show-task --watch to continue doing what the run CLI would have done, ie watch the task, print any log messages, and exit when the task completes. Or without --watch, show-task prints the current state of the task at that time. In both cases, the exit code is 0 if the show-task command ran successfully. So it’s from a consistency perspective that the action error code is the YAML - it’s considered part of the action results rather than an indication that the command to queue the action ran ok or not.

The idea about --ignore-missing is interesting. We would want to still include in any result an indication that the action was not found, but treat as a no-op with 0 return code.