How to use Juju to debug logs

doc
1

Juju’s debug-log feature is one of the most powerful tools you have as a Juju developer. It is extremely useful for debugging and finding the root cause of an issue.

There are two steps to successfully using debug logs:

  • Set the logging-config
  • Use juju debug-log

Setting logging-config

Juju saves or discards logs according to the value of the model config key logging-config. Therefore, logging-config needs to be set before the events you want to collect logs for (i.e. before attempting to reproduce a bug).

Note: since logging-config is a model config key, it is scoped per model. You may have to set logging-config on the controller model as well as any workload models you’re interested in.

The general form of the logging-config key is:

filter1=level1;...;filterN=levelN

Filters

filterX can be any of the following:

  • <root> - matches all machine agent logs
  • unit - matches all unit agent logs
  • a module name, e.g. juju.worker.apiserver
  • a label, e.g. #charmhub

A module represents a single component of Juju, e.g. a worker. Generally, modules correspond one-to-one with Go packages in the Juju source tree. The module name is the value passed to loggo.GetLogger or loggo.GetLoggerWithLabels.

Modules have a nested tree structure - for example, the juju.api module includes submodules juju.api.application, juju.api.cloud, etc. <root> is the root of this module tree.

Labels cut across the module tree, grouping various modules which deal with a certain feature or information flow. For example, the #charmhub label includes all modules involved in making a request to Charmhub.

Labels are defined in core/logger/labels.go. The currently supported labels are:

Label Description
#http HTTP requests
#metrics Metric outputs - use as a fallback when Prometheus isn’t available
#charmhub Charmhub client and callers.
#cmr Cross model relations
#cmr-auth Authentication for cross model relations
#secrets Juju secrets

Levels

There are six severity levels in Juju. In decreasing order of severity:

Level Description
CRITICAL Indicates a severe failure which could bring down the system.
ERROR Indicates failure to complete a routine operation.
WARNING Indicates something is not as expected, but this is not necessarily going to cause an error.
INFO A regular log message intended for the user.
DEBUG Information intended to assist developers in debugging.
TRACE The lowest level - includes the full details of input args, return values, HTTP requests sent/received, etc.

When you set logging-config to module=level, then Juju saves that module’s logs for the given severity level and above. For example, setting logging-config to juju.worker.uniter=WARNING will capture all CRITICAL, ERROR and WARNING logs for the uniter, but discard logs for lower severity levels (INFO, DEBUG, TRACE).

Examples

To collect debug logs for the dbaccessor worker:

juju model-config -m controller logging-config="juju.worker.dbaccessor=DEBUG"

To collect debug logs for the mysql/0 unit:

juju model-config -m foo logging-config="unit.mysql/0=DEBUG"

To collect trace logs for Charmhub requests:

juju model-config -m controller logging-config="#charmhub=TRACE"

Using juju debug-log

After setting the logging-config, you can view the logs using the juju debug-log command. debug-log has several different --include* / --exclude* flags that you can use to filter the logs and extract the relevant log lines. These options are pretty well-covered in juju debug-log --help, but in short:

  • --include / --exclude filters by entity (machine, unit, or application)
  • --include-module / --exclude-module filters by module
  • --include-label / --exclude-label filters by label

The individual --include* / --exclude* values are ORed together, and then ANDed together for each different --include* / --exclude* flag.

See also: