See also: Log
Manage the logs
Stream the logs
If you want to control the amount of logging detail first, see Manage the logging configuration.
To stream the logs in the current model, run the debug-log command:
juju debug-log
In a machine deployment, this will output logs for all the Juju machine and unit agents, starting with the last 10 lines, in the following format:
<entity (machine or unit)> <timestamp> <log level> <Juju module> <message>
In Kubernetes deployments, this will not show any logs below the model level.
See also: Kubernetes agent logs
The command has various options that allow you to control the length, appearance, amount and type of detail.
Expand to see examples
To begin with the last 30 log messages:
juju debug-log -n 30
To begin with all the log messages:
juju debug-log --replay
To begin with the last twenty log messages for the ‘lxd-pilot’ model:
juju debug-log -m lxd-pilot -n 20
To begin with the last 500 lines. The grep utility is used as a text filter:
juju debug-log -n 500 | grep amd64
To begin with the last 1000 lines and exclude messages from machine 3:
juju debug-log -n 1000 --exclude machine-3
To select all the messages emitted from a particular unit and a particular machine in the entire log:
juju debug-log --replay --include unit-mysql-0 --include machine-1
The unit can also be written ‘mysql/0’ (as shown by juju status).
To see all WARNING and ERROR messages in the entire log:
juju debug-log --replay --level WARNING
To see all logs on the cmr topic (label):
juju debug-log --include-label cmr
To progressively exclude more content from the entire log:
juju debug-log --replay --exclude-module juju.state.apiserver
juju debug-log --replay --exclude-module juju.state
juju debug-log --replay --exclude-module juju
To begin with the last 2000 lines and include messages pertaining to both the juju.cmd and juju.worker modules:
juju debug-log --lines 2000 \
--include-module juju.cmd \
--include-module juju.worker
See more:
juju debug-log
Configure the logging level
Juju saves or discards logs according to the value of the model config key logging-config. Therefore, logging-config needs to be set before the events you want to collect logs for (i.e. before attempting to reproduce a bug).
Set values.
- To change the logging configuration for machine and unit agents:
Run themodel-configcommand with thelogging-configkey set to a"-enclosed, semi-colon-separated list of<filter>=<verbosity level>pairs.
Example setting machine agent logs to `WARNING` and unit agent logs to `TRACE`
juju model-config logging-config="<root>=WARNING;unit=TRACE"
Example setting unit agent logs for unit `0` of `mysql` to `DEBUG`
juju model-config logging-config="unit.mysql/0=DEBUG"
See more: How to configure a model, List of model configuration keys >
logging-config
To avoid filling up the database unnecessarily:
When verbose logging is no longer needed, return logging to normal levels!
- To change the logging configuration on a per-unit-agent basis:
- SSH into the unit’s machine. E.g., for
mysql/0:
juju ssh mysql/0
- Open the unit’s agent configuration file. For our example, it will be
/var/lib/juju/agents/unit-mysql-0/agent.conf/ Then, find thevaluessection, and add a line with the fieldLOGGING_OVERRIDEset tojuju=<desired log level>, belowTRACE. The bottom of the file should now look as below:
loggingconfig: <root>=WARNING;unit=DEBUG
values:
CONTAINER_TYPE: ""
NAMESPACE: ""
LOGGING_OVERRIDE: juju=trace
mongoversion: "0.0"
- Restart the affected agent:
sudo systemctl restart jujud-unit-mysql-0.service
Get values. To verify the current logging configuration for machine and unit agents, run model-config followed by the logging-config key:
juju model-config logging-config
Sample output:
<root>=WARNING;unit=DEBUG;#http=TRACE
which means that the machine agent (<root>) log level is set to WARNING, the unit agent (unit) log level is set at DEBUG, and the http label is set to TRACE.
See more: How to configure a model, List of model configuration keys >
logging-config
Forward logs to an external logsink
You can optionally forward log messages to a remote syslog server over a secure TLS connection, on a per-model basis, as below:
See Rsyslog documentation for help with security-related files (certificates, keys) and the configuration of the remote syslog server.
- Configure the controller for remote logging by configuring it during controller creation as below:
juju bootstrap <cloud> --config mylogconfig.yaml
where the YAML file is as below:
syslog-host: <host>:<port>
syslog-ca-cert: |
-----BEGIN CERTIFICATE-----
<cert-contents>
-----END CERTIFICATE-----
syslog-client-cert: |
-----BEGIN CERTIFICATE-----
<cert-contents>
-----END CERTIFICATE-----
syslog-client-key: |
-----BEGIN PRIVATE KEY-----
<cert-contents>
-----END PRIVATE KEY-----
See more: How to configure a controller
- Enable log forwarding for a model by configuring it as below:
juju model-config -m <model> logforward-enabled=True
An initial 100 (maximum) existing log lines will be forwarded.
See more: How to configure a model
Pro tip: You can configure remote logging and enable log forwarding on all the controller’s models in one step by running
juju bootstrap <cloud> --config mylogconfig.yaml --config logforward-enabled=True
Manage the log files
Only applicable for machines – for Kubernetes logs are written directly to stdout of the container and can be retrieved with native Kubernetes methods, e.g., kubectl logs -c <container-name> <pod-name> -n <model-name> .
See more: Juju agent logs – machines, Juju agent logs – Kubernetes
View the log files
To view the Juju log files in a Juju machine:
- Open a shell into the machine:
(1a) IfJuju can connect to the machine (i.e., the output contains Connected to <IP address>) and the machine is fully provisioned, use juju ssh. For example, to connect to machine 0:
juju ssh 0
(1b) IfJuju can connect to the machine (i.e., the output contains Connected to <IP address>) but the machine is not fully provisioned (e.g., command hangs at Running machine configuration script...), use the ssh command followed by the address of the machine and the path to the place where Juju stores your SSH keys (including the ones it generates automatically for you):
ssh ubuntu@<ip-address> -i <juju-data-dir>/ssh/juju_id_rsa
Here, <juju-data-dir> defaults to ~/.local/share/juju, but if you’ve set the JUJU_DATA environment variable, it will be equal to that instead.
(1c) If Juju cannot connect to the machine (i.e., the command never reaches Connected to <IP address>), use cloud-specific tools. For example, for the LXD cloud:
lxc exec [container-name] bash
or, for the MicroK8s cloud:
microk8s kubectl exec controller-0 -itc api-server -n [namespace] -- bash
See more: How to access a machine via SSH
- Examine the log files under
/var/logwith commands such ascat,less, ortail -f, for example:
cat /var/log/juju
Which log to look at depends on the type of failure, but generally speaking, syslog, cloud-init.log, cloud-init-output.log, and /var/log/juju are good ones to look at.
Expand to see an example for a controller machine
# SSH into machine 0 of the controller model:
juju ssh -m controller 0
# Navigate to the logs directory:
cd /var/log/juju
# List the contents:
ls
# View, e.g., the audit log file:
cat audit.log
Control the log file rotation
See also: List of controller configuration keys
Juju has settings to control the rotation of the various log files it produces.
There are two settings for each log file type: maximum size and number of backups. When the current log file of a particular type reaches the maximum size, Juju renames the log file to include a timestamp and gzips it, producing a “backup” log file.
Here’s an example of the controller’s machine agent logs with the maximum size set to 1MB, showing two timestamped backups as well as the current log file:
$ juju bootstrap localhost --config agent-logfile-max-size=1MB
$ lxc exec juju-6bf629-0 -- ls -l /var/log/juju
...
-rw-r----- 1 syslog adm 3577 Jan 12 02:01 machine-0-2022-01-12T02-01-07.995.log.gz
-rw-r----- 1 syslog adm 3578 Jan 12 02:02 machine-0-2022-01-12T02-02-08.011.log.gz
-rw-r----- 1 syslog adm 600000 Jan 12 02:02 machine-0.log
The full list of the controller settings that configure log file rotation is shown below. Normally these are set at bootstrap time with the --config option (see How to configure a controller).
- The following config settings configure agent log files, including the API server “log sink”, the machine agent logs on controller and unit machines, and the unit agent logs:
agent-logfile-max-backupsagent-logfile-max-size
- The following config settings configure the audit log files (note the missing “file” in the key name compared to the agent log file settings):
audit-log-max-backupsaudit-log-max-size
- The following config settings configure the model log files:
model-logfile-max-backupsmodel-logfile-max-size
Contributors: @hmlanigan, @jameinel, @manadart, @tmihoc
