N48 Juju logs

This page outlines the different types of logs that a Juju controller generates and provides information on how Juju users can access them. The following sections cover:

Logging levels

Juju uses both hierarchical logging (each subsystem has a named logger which can be configured independently of the other loggers) and level-based logging (a severity level is specified for each log entry). In particular, Juju supports the following log severity levels (listed from most verbose to least verbose):

  • TRACE
  • DEBUG
  • INFO
  • WARNING
  • ERROR

Logging may be configured on a per-model basis via the logging-config option. The following command will display the currently active logging configuration:

juju model-config logging-config

Output will be similar to the following:

<root>=WARNING;unit=DEBUG

The above is the default configuration. It sets the Juju machine agent log severity level at ‘WARNING’ and the unit agent log level at ‘DEBUG’.

Changing the log level settings

When attempting to diagnosing an issue it is usually beneficial to increase the verbosity of the Juju logs. For instance, the following command ensures that all Juju loggers log at WARNING level with the exception of the Juju unit agents which should log at TRACE level:

juju model-config logging-config="<root>=WARNING;unit=TRACE"

As log entries are stored by the Juju controller in its database, it is recommended to eventually reset log settings to their default value so as to avoid filling up the database when verbose logging is no longer required.

Overriding logging configuration for a specific unit

As we’ve seen, the logging level for machine agents and unit agents are specified as a single model configuration setting. However, in some situations (e.g. targeted verbose debugging) it may be desirable to increase the logging level on a per-agent basis. This can also be done after having reduced the model-wide log level (as explained in the previous section).

In the following example, the goal is to override the log settings for an individual unit of the MySQL application so that it always logs at TRACE level. Assuming that the unit we are interested in is deployed to machine 0, the first step is to establish an ssh connection to the machine:

juju ssh mysql/0

To override the config settings for the unit, edit /var/lib/juju/agents/unit-mysql-0/agent.conf and append LOGGING_OVERRIDE: juju=trace to the end of the values section.

The modified file should looks as follows:

loggingconfig: <root>=WARNING;unit=DEBUG
values:
  CONTAINER_TYPE: ""
  NAMESPACE: ""
  LOGGING_OVERRIDE: juju=trace
mongoversion: "0.0"

Finally, the machine agent will need to be restarted for this change to take effect:

sudo systemctl restart jujud-machine-0.service

As of Juju 2.9, machine and unit agents run as a single process. If you are working with a pre-2.9 Juju controller, you must instead restart the unit agent service as follows:

bash sudo systemctl restart jujud-unit-mysql0.service

Log files

Agent logs (for the machine and any units deployed to the machine) are stored locally on each provisioned machine and can be accessed by establishing an ssh connection to a machine and inspecting the contents of the /var/log/juju folder:

juju ssh 2 ls -lh /var/log/juju

Output:

-rw------- 1 syslog syslog  22K Apr 28 00:42 machine-2.log
-rw------- 1 syslog syslog 345K Apr 28 16:58 unit-nfs2-0.log

Running the above command on a controller machine would reveal an additional log file named logsink.log. Each Juju agent ships a copy of the local logs to the controller which is responsible for appending them to logsink.log and storing them to the controller’s database for query purposes.

In a High availability scenario, logsink.log is not guaranteed to contain all messages since agents have a choice of several controllers to send their logs to. The debug-log command should be used for accessing consolidated data across all controllers.

Accessing logs for a particular model

As mentioned in the previous section, the Juju agents running on provisioned machines ship any generated log entries to the controller which then tags them with the model ID they originated from and stores them into Juju’s database. The juju debug-log command can be used to query the log entries for a particular model.

See Controller HA and logging when viewing logs in an HA context.

The debug-log command

The debug-log command shows the consolidated logs for all Juju agents that are part of a Juju model. By default, juju debug-log will render the log contents for the current model. However, the -m or --model flag may be specified to display the logs for the specified model.

Not sure which is the currently active model?: You can either run the juju whoami or juju status command to find the name of the current model.

The command will first render a fixed number of existing log lines (10 lines by default; count may be adjusted via the --lines flag) and then enter log streaming mode where the client waits for new log entries to appear and renders them as they become available. As this is a blocking command, Ctrl+C must be pressed to stop tailing the logs and return back to the shell.

In some cases, instead of the last N lines of logs, you may want to render the full set of logged messages. This can be achieved by specifying the --replay flag:

juju debug-log --replay

Filtering log output

Each Juju log line consists of the following sections:

<entity> <timestamp> <log-level> <module>:<line-no> <message>

The output of the debug-log command may be filtered by specifying any combination of the supported filter flags:

  • The --include and --exclude options select and deselect, respectively, the entity that logged the message. An entity is a Juju machine or unit agent. The entity names are similar to the names shown by juju status.
  • The --include-module and --exclude-module options can be used to influence the type of message displayed based on a (dotted) module name.
  • The --level option places a limit on logging verbosity (e.g. --level INFO will allow messages of levels ‘INFO’, ‘WARNING’, and ‘ERROR’ to be shown).

Here are some examples for the advanced log filtering functionality:

Log messages but filter those that originated from machine 3:

juju debug-log --exclude machine-3

Display logs that were generated by unit 0 of mysql on machine 1:

juju debug-log --include mysql/0 --include machine-1

Replay all log entries but only display those at WARNING level or higher:

juju debug-log --replay --level WARNING

Use dotted notation to progressively exclude more and more content from the entire set of logs:

juju debug-log --replay --exclude-module juju.state.apiserver
juju debug-log --replay --exclude-module juju.state
juju debug-log --replay --exclude-module juju

Display messages emitted by the juju.cmd and juju.worker modules:

juju debug-log \
    --include-module juju.cmd \
    --include-module juju.worker

Forwarding logs using rsyslog

On a per-model basis, log messages can optionally be forwarded to a remote syslog server over a secure TLS connection.

See Rsyslog documentation for help with security-related files (certificates, keys) and the configuration of the remote syslog server.

Configuring log forwarding

Log forwarding can be configured at bootstrap time by supplying a YAML-formatted configuration file with the appropriate configuration settings:

juju bootstrap <cloud> --config logconfig.yaml

The YAML file should contain the following entries:

syslog-host: <host>:<port>
syslog-ca-cert: |
-----BEGIN CERTIFICATE-----
 <cert-contents>
-----END CERTIFICATE-----
syslog-client-cert: |
-----BEGIN CERTIFICATE-----
 <cert-contents>
-----END CERTIFICATE-----
syslog-client-key: |
-----BEGIN PRIVATE KEY-----
 <cert-contents>
-----END PRIVATE KEY-----

Enabling log forwarding for a specific model

After configuring the controller with the required log forwarding settings, log forwarding can be enabled for a particular model using the following command:

juju model-config -m <model> logforward-enabled=True

Enabling log forwarding for a all models

It is also possible to configure and enable forwarding for all models at bootstrap time using the following command:

juju bootstrap <cloud> --config logforward-enabled=True --config logconfig.yaml

Accessing the audit logs

Juju audit logging maintains a time-ordered event stream which includes all commands that Juju users ran every Juju model managed by the controller. These logs can be accessed by establishing an ssh connection to the controller machine and inspecting the contents of the /var/log/juju/audit.log file.

The audit log entries are encoded as JSON documents and include:

  • Conversations: a collection of API methods associated with a single top-level CLI command (e.g. juju deploy).
  • Request : API method invocations.
  • ResponseErrors: errors raised during the invocation of an API method.

Information can be filtered out of the audit log to prevent its file(s) from growing without bounds and making it difficult to read. See Excluding information from the audit log.