How to manage logs

doc
#1

See also: Juju logs

Contents:

Examine the log files directly

To view the Juju log files in a Juju machine:

  1. SSH into the machine:
juju ssh <machine ID>
  1. Navigate to /var/log/juju:
cd /var/log/juju
  1. List the contents:
ls
  1. Open a file:
cat <filename>
Expand to see an example for a controller machine 
# SSH into machine 0 of the controller model:
juju ssh -m controller 0

# Navigate to the logs directory:
cd /var/log/juju

# List the contents:
ls

# View, e.g., the audit log file:
cat audit.log

Stream all the logs in a model

If you want to control the amount of logging detail first, see Manage the logging configuration.

To stream the logs in the current model, run the debug-log command:

juju debug-log

This will output logs for all the Juju (machine and unit) agents, starting with the last 10 lines, in the following format:

<entity (machine or unit)> <timestamp> <log level> <Juju module> <message>

The command has various options that allow you to control the length, appearance, amount and type of detail.

Expand to see examples

To begin with the last 30 log messages:

juju debug-log -n 30

To begin with all the log messages:

juju debug-log --replay

To begin with the last twenty log messages for the ‘lxd-pilot’ model:

juju debug-log -m lxd-pilot -n 20

To begin with the last 500 lines. The grep utility is used as a text filter:

juju debug-log -n 500 | grep amd64

To begin with the last 1000 lines and exclude messages from machine 3:

juju debug-log -n 1000 --exclude machine-3

To select all the messages emitted from a particular unit and a particular machine in the entire log:

juju debug-log --replay --include unit-mysql-0 --include machine-1

The unit can also be written ‘mysql/0’ (as shown by juju status).

To see all WARNING and ERROR messages in the entire log:

juju debug-log --replay --level WARNING

To see all logs on the cmr topic (label):

juju debug-log --include-label cmr

To progressively exclude more content from the entire log:

juju debug-log --replay --exclude-module juju.state.apiserver
juju debug-log --replay --exclude-module juju.state
juju debug-log --replay --exclude-module juju

To begin with the last 2000 lines and include messages pertaining to both the juju.cmd and juju.worker modules:

juju debug-log --lines 2000 \
    --include-module juju.cmd \
    --include-module juju.worker

See more: juju debug-log

Manage the logging configuration

Juju saves or discards logs according to the value of the model config key logging-config. Therefore, logging-config needs to be set before the events you want to collect logs for (i.e. before attempting to reproduce a bug).

Set values.

  • To change the logging configuration for machine and unit agents:
    Run the model-config command with the logging-config key set to a "-enclosed, semi-colon-separated list of <filter>=<verbosity level> pairs. For example:
juju model-config logging-config="<root>=WARNING;unit=TRACE"

See more: How to configure a model, List of model configuration keys > logging-config

To avoid filling up the database unnecessarily:
When verbose logging is no longer needed, return logging to normal levels!

  • To change the logging configuration on a per-unit-agent basis:
  1. SSH into the unit’s machine. E.g., for mysql/0:
juju ssh mysql/0
  1. Open the unit’s agent configuration file. For our example, it will be /var/lib/juju/agents/unit-mysql-0/agent.conf/ Then, find the values section, and add a line with the field LOGGING_OVERRIDE set to juju=<desired log level>, below TRACE. The bottom of the file should now look as below:
loggingconfig: <root>=WARNING;unit=DEBUG
values:
  CONTAINER_TYPE: ""
  NAMESPACE: ""
  LOGGING_OVERRIDE: juju=trace
mongoversion: "0.0"
  1. Restart the affected agent:
sudo systemctl restart jujud-unit-mysql-0.service

Get values. To verify the current logging configuration for machine and unit agents, run model-config followed by the logging-config key:

juju model-config logging-config

Sample output:

<root>=WARNING;unit=DEBUG;#http=TRACE

which means that the machine agent (<root>) log level is set to WARNING, the unit agent (unit) log level is set at DEBUG, and the http label is set to TRACE.

See more: How to configure a model, List of model configuration keys > logging-config

Configure the log file rotation

See also: List of controller configuration keys

Juju has settings to control the maximum size and rotation of the various log files it produces.

There are two settings for each log file type: maximum size, and number of backups. When the current log file of a particular type reaches the maximum size, Juju renames the log file to include a timestamp and gzips it, producing a “backup” log file.

Here’s an example of the controller’s machine agent logs with the maximum size set to 1MB, showing two timestamped backups as well as the current log file:

$ juju bootstrap localhost --config agent-logfile-max-size=1MB
$ lxc exec juju-6bf629-0 -- ls -l /var/log/juju
...
-rw-r----- 1 syslog adm   3577 Jan 12 02:01 machine-0-2022-01-12T02-01-07.995.log.gz
-rw-r----- 1 syslog adm   3578 Jan 12 02:02 machine-0-2022-01-12T02-02-08.011.log.gz
-rw-r----- 1 syslog adm 600000 Jan 12 02:02 machine-0.log

The full list of the controller settings that configure log file rotation is shown below. Normally these are set at bootstrap time with the --config option (see How to configure a controller).

Configuring agent log file roation

The following config settings configure agent log files, including the API server “log sink”, the machine agent logs on controller and unit machines, and the unit agent logs:

  • agent-logfile-max-backups
  • agent-logfile-max-size

Configuring audit log file rotation

The following config settings configure the audit log files (note the missing “file” in the key name compared to the agent log file settings):

  • audit-log-max-backups
  • audit-log-max-size

Configuring model log file rotation

The following config settings configure the model log files:

  • model-logfile-max-backups
  • model-logfile-max-size

Forward logs to an external logsink

You can optionally forward log messages to a remote syslog server over a secure TLS connection, on a per-model basis, as below:

See Rsyslog documentation for help with security-related files (certificates, keys) and the configuration of the remote syslog server.

  1. Configure the controller for remote logging by configuring it during controller creation as below:
juju bootstrap <cloud> --config mylogconfig.yaml

where the YAML file is as below:

syslog-host: <host>:<port>
syslog-ca-cert: |
-----BEGIN CERTIFICATE-----
 <cert-contents>
-----END CERTIFICATE-----
syslog-client-cert: |
-----BEGIN CERTIFICATE-----
 <cert-contents>
-----END CERTIFICATE-----
syslog-client-key: |
-----BEGIN PRIVATE KEY-----
 <cert-contents>
-----END PRIVATE KEY-----

See more: How to configure a controller

  1. Enable log forwarding for a model by configuring it as below:
juju model-config -m <model> logforward-enabled=True

An initial 100 (maximum) existing log lines will be forwarded.

See more: How to configure a model

Pro tip: You can configure remote logging and enable log forwarding on all the controller’s models in one step by running

juju bootstrap <cloud> --config mylogconfig.yaml --config logforward-enabled=True
Log
How to use Juju to debug logs
Command 'juju debug-log'
OLM How-to guides
Juju OLM Documentation