This page outlines the different types of logs that a Juju controller generates and provides information on how Juju users can access them. The following sections cover:
- Logging levels
- Accessing logs for a particular model
- Forwarding logs using rsyslog
- Accessing the audit logs
Juju uses both hierarchical logging (each subsystem has a named logger which can be configured independently of the other loggers) and level-based logging (a severity level is specified for each log entry). In particular, Juju supports the following log severity levels (listed from most verbose to least verbose):
Logging may be configured on a per-model basis via the
logging-config option. The following command will display the currently active logging configuration:
juju model-config logging-config
Output will be similar to the following:
The above is the default configuration. It sets the Juju machine agent log severity level at ‘WARNING’ and the unit agent log level at ‘DEBUG’.
Changing the log level settings
When attempting to diagnosing an issue it is usually beneficial to increase the verbosity of the Juju logs. For instance, the following command ensures that all Juju loggers log at
WARNING level with the exception of the Juju unit agents which should log at
juju model-config logging-config="<root>=WARNING;unit=TRACE"
As log entries are stored by the Juju controller in its database, it is recommended to eventually reset log settings to their default value so as to avoid filling up the database when verbose logging is no longer required.
Overriding logging configuration for a specific unit
As we’ve seen, the logging level for machine agents and unit agents are specified as a single model configuration setting. However, in some situations (e.g. targeted verbose debugging) it may be desirable to increase the logging level on a per-agent basis. This can also be done after having reduced the model-wide log level (as explained in the previous section).
In the following example, the goal is to override the log settings for an individual unit of the MySQL application so that it always logs at
TRACE level. Assuming that the unit we are interested in is deployed to machine
0, the first step is to establish an ssh connection to the machine:
juju ssh mysql/0
To override the config settings for the unit, edit
/var/lib/juju/agents/unit-mysql-0/agent.conf and append
LOGGING_OVERRIDE: juju=trace to the end of the
The modified file should looks as follows:
loggingconfig: <root>=WARNING;unit=DEBUG values: CONTAINER_TYPE: "" NAMESPACE: "" LOGGING_OVERRIDE: juju=trace mongoversion: "0.0"
Finally, the machine agent will need to be restarted for this change to take effect:
sudo systemctl restart jujud-machine-0.service
As of Juju 2.9, machine and unit agents run as a single process. If you are working with a pre-2.9 Juju controller, you must instead restart the unit agent service as follows:
bash sudo systemctl restart jujud-unit-mysql0.service
Agent logs (for the machine and any units deployed to the machine) are stored locally on each provisioned machine and can be accessed by establishing an ssh connection to a machine and inspecting the contents of the
juju ssh 2 ls -lh /var/log/juju
-rw------- 1 syslog syslog 22K Apr 28 00:42 machine-2.log -rw------- 1 syslog syslog 345K Apr 28 16:58 unit-nfs2-0.log
Running the above command on a controller machine would reveal an additional log file named
logsink.log. Each Juju agent ships a copy of the local logs to the controller which is responsible for appending them to
logsink.log and storing them to the controller’s database for query purposes.
In a High availability scenario,
logsink.log is not guaranteed to contain all messages since agents have a choice of several controllers to send their logs to. The
debug-log command should be used for accessing consolidated data across all controllers.
Accessing logs for a particular model
As mentioned in the previous section, the Juju agents running on provisioned machines ship any generated log entries to the controller which then tags them with the model ID they originated from and stores them into Juju’s database. The
juju debug-log command can be used to query the log entries for a particular model.
See Controller HA and logging when viewing logs in an HA context.
The debug-log command
debug-log command shows the consolidated logs for all Juju agents that are part of a Juju model. By default,
juju debug-log will render the log contents for the current model. However, the
--model flag may be specified to display the logs for the specified model.
Not sure which is the currently active model?:
You can either run the
juju whoami or
juju status command to find the name of the current model.
The command will first render a fixed number of existing log lines (10 lines by default; count may be adjusted via the
--lines flag) and then enter log streaming mode where the client waits for new log entries to appear and renders them as they become available. As this is a blocking command,
Ctrl+C must be pressed to stop tailing the logs and return back to the shell.
In some cases, instead of the last N lines of logs, you may want to render the full set of logged messages. This can be achieved by specifying the
juju debug-log --replay
Filtering log output
Each Juju log line consists of the following sections:
<entity> <timestamp> <log-level> <module>:<line-no> <message>
The output of the
debug-log command may be filtered by specifying any combination of the supported filter flags:
--excludeoptions select and deselect, respectively, the entity that logged the message. An entity is a Juju machine or unit agent. The entity names are similar to the names shown by
--exclude-moduleoptions can be used to influence the type of message displayed based on a (dotted) module name.
--leveloption places a limit on logging verbosity (e.g.
--level INFOwill allow messages of levels ‘INFO’, ‘WARNING’, and ‘ERROR’ to be shown).
Here are some examples for the advanced log filtering functionality:
Log messages but filter those that originated from machine 3:
juju debug-log --exclude machine-3
Display logs that were generated by unit 0 of mysql on machine 1:
juju debug-log --include mysql/0 --include machine-1
Replay all log entries but only display those at WARNING level or higher:
juju debug-log --replay --level WARNING
Use dotted notation to progressively exclude more and more content from the entire set of logs:
juju debug-log --replay --exclude-module juju.state.apiserver juju debug-log --replay --exclude-module juju.state juju debug-log --replay --exclude-module juju
Display messages emitted by the juju.cmd and juju.worker modules:
juju debug-log \ --include-module juju.cmd \ --include-module juju.worker
Forwarding logs using rsyslog
On a per-model basis, log messages can optionally be forwarded to a remote syslog server over a secure TLS connection.
See Rsyslog documentation for help with security-related files (certificates, keys) and the configuration of the remote syslog server.
Configuring log forwarding
Log forwarding can be configured at bootstrap time by supplying a YAML-formatted configuration file with the appropriate configuration settings:
juju bootstrap <cloud> --config logconfig.yaml
The YAML file should contain the following entries:
syslog-host: <host>:<port> syslog-ca-cert: | -----BEGIN CERTIFICATE----- <cert-contents> -----END CERTIFICATE----- syslog-client-cert: | -----BEGIN CERTIFICATE----- <cert-contents> -----END CERTIFICATE----- syslog-client-key: | -----BEGIN PRIVATE KEY----- <cert-contents> -----END PRIVATE KEY-----
Enabling log forwarding for a specific model
After configuring the controller with the required log forwarding settings, log forwarding can be enabled for a particular model using the following command:
juju model-config -m <model> logforward-enabled=True
Enabling log forwarding for a all models
It is also possible to configure and enable forwarding for all models at bootstrap time using the following command:
juju bootstrap <cloud> --config logforward-enabled=True --config logconfig.yaml
Accessing the audit logs
Juju audit logging maintains a time-ordered event stream which includes all commands that Juju users ran every Juju model managed by the controller. These logs can be accessed by establishing an ssh connection to the controller machine and inspecting the contents of the
The audit log entries are encoded as JSON documents and include:
Conversations: a collection of API methods associated with a single top-level CLI command (e.g.
- Request : API method invocations.
- ResponseErrors: errors raised during the invocation of an API method.
Information can be filtered out of the audit log to prevent its file(s) from growing without bounds and making it difficult to read. See Excluding information from the audit log.