Overview
Duration: 3:00
Before you get started!
Welcome to the world of operators! In this series of tutorials, we will walk you through all the necessary steps to explore and learn operators through some basic examples. Starting with just your laptop, you will learn how to use operators for sophisticated application management at scale, across multiple Kubernetes clusters, cloud virtual machines and ultimately bare metal.
This tutorial assumes you have successfully completed the previous tutorial - Bundles.
What are operators?
Operators are pieces of software designed to drive other software. They encapsulate the operational code of applications, shifting the burden of lifecycle management from configuration files and manual steps to an operator instance. Operators are a form of automation that is gaining popularity in the Kubernetes domain where traditional configuration management does not work. However, operators work with traditional virtual and bare metal machines as well.
What is Juju OLM?
Juju is a universal Operator Lifecycle Manager (OLM) which provides services to operators. It provides resources for operators, deploys them, manages their lifecycle, delivers configuration updates, etc. Juju OLM is universal, meaning that it supports container substrates like Kubernetes as well as traditional machine substrates like bare metal, VMware, OpenStack, or public cloud instances.
In this tutorial you will learn how to:
- Interrogate logging output for both the application and operator workloads
- View and change logging level for applications
- View and change logging level for operators
You will only need:
- A machine with 8 GB of RAM
- Microk8s and Juju installed
- Juju controller bootstrapped on MicroK8s
- The Prometheus and Grafana operators deployed to Microk8s
Application logging
Duration: 5:00
Operators provide capabilities to change the logging level for applications and interrogate applications logging output.
In the following steps, we will change the logging level for the Prometheus application with an operator and see how to interrogate the logs using kubectl command.
First let’s see what the current status of the model is:
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
monitoring mk8s microk8s/localhost 2.9.10 unsupported 09:26:31+01:00
App Version Status Scale Charm Store Channel Rev OS Address Message
prometheus-k8s prom/prometheus:latest waiting 1 prometheus-k8s charmhub stable 1 kubernetes 10.152.183.44
Unit Workload Agent Address Ports Message
prometheus-k8s/0* waiting idle 10.1.98.30 9090/TCP waiting for container
The logging level for an application is defined in its operator configuration options. As we remember from the previous tutorials, the configuration options of an operator can be displayed using the following command:
$ juju config prometheus-k8s
application: prometheus
...
log-level:
description: |
Prometheus server log level (only log messages with the given severity
or above). Must be one of: [debug, info, warn, error, fatal].
If not set, the Prometheus default one (info) will be used.
source: unset
type: string
...
Looking at log-level config option allows the updating of the logging output a running prometheus application. We can see this option in action by looking at the logging output from the application pod. Run the following command in a new terminal window, so you can watch the output:
$ microk8s.kubectl logs -n monitoring prometheus-0
level=info ts=2021-07-21T08:26:21.481Z caller=main.go:443 msg="Starting Prometheus" version="(version=2.28.1, branch=HEAD, revision=b0944590a1c9a6b35dc5a696869f75f422b107a1)"
level=info ts=2021-07-21T08:26:21.482Z caller=main.go:448 build_context="(go=go1.16.5, user=root@2915dd495090, date=20210701-15:20:10)"
level=info ts=2021-07-21T08:26:21.482Z caller=main.go:449 host_details="(Linux 5.4.0-77-generic #86-Ubuntu SMP Thu Jun 17 02:35:03 UTC 2021 x86_64 prometheus-k8s-0 (none))"
level=info ts=2021-07-21T08:26:21.482Z caller=main.go:450 fd_limits="(soft=65536, hard=65536)"
level=info ts=2021-07-21T08:26:21.482Z caller=main.go:451 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2021-07-21T08:26:21.484Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2021-07-21T08:26:21.484Z caller=main.go:824 msg="Starting TSDB ..."
level=info ts=2021-07-21T08:26:21.486Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
level=info ts=2021-07-21T08:26:21.490Z caller=head.go:780 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2021-07-21T08:26:21.490Z caller=head.go:794 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.475µs
level=info ts=2021-07-21T08:26:21.490Z caller=head.go:800 component=tsdb msg="Replaying WAL, this may take a while"
level=info ts=2021-07-21T08:26:21.490Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2021-07-21T08:26:21.490Z caller=head.go:860 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.179µs wal_replay_duration=363.361µs total_replay_duration=415.731µs
level=info ts=2021-07-21T08:26:21.491Z caller=main.go:851 fs_type=EXT4_SUPER_MAGIC
level=info ts=2021-07-21T08:26:21.491Z caller=main.go:854 msg="TSDB started"
level=info ts=2021-07-21T08:26:21.491Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2021-07-21T08:26:21.492Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=869.925µs remote_storage=1.653µs web_handler=351ns query_engine=862ns scrape=447.702µs scrape_sd=26.31µs notify=902ns notify_sd=1.763µs rules=1.122µs
level=info ts=2021-07-21T08:26:21.492Z caller=main.go:796 msg="Server is ready to receive web requests."
Not surprisingly all log messages are of the “info” level as this is the default value according to the configuration option description.
Now let’s change the value of the log-level option to “debug” and see that it has been updated:
$ juju config prometheus-k8s log-level=debug
$ juju config prometheus-k8s log-level
debug
After a short period of time, the logs from the pod should include debug messages:
$ microk8s.kubectl logs -n monitoring prometheus-k8s-0
level=info ts=2021-07-21T08:32:55.867Z caller=main.go:443 msg="Starting Prometheus" version="(version=2.28.1, branch=HEAD, revision=b0944590a1c9a6b35dc5a696869f75f422b107a1)"
level=info ts=2021-07-21T08:32:55.867Z caller=main.go:448 build_context="(go=go1.16.5, user=root@2915dd495090, date=20210701-15:20:10)"
level=info ts=2021-07-21T08:32:55.867Z caller=main.go:449 host_details="(Linux 5.4.0-77-generic #86-Ubuntu SMP Thu Jun 17 02:35:03 UTC 2021 x86_64 prometheus-k8s-0 (none))"
level=info ts=2021-07-21T08:32:55.867Z caller=main.go:450 fd_limits="(soft=65536, hard=65536)"
level=info ts=2021-07-21T08:32:55.867Z caller=main.go:451 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2021-07-21T08:32:55.870Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2021-07-21T08:32:55.870Z caller=main.go:824 msg="Starting TSDB ..."
level=info ts=2021-07-21T08:32:55.872Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
level=info ts=2021-07-21T08:32:55.874Z caller=head.go:780 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2021-07-21T08:32:55.874Z caller=head.go:794 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.705µs
level=info ts=2021-07-21T08:32:55.874Z caller=head.go:800 component=tsdb msg="Replaying WAL, this may take a while"
level=info ts=2021-07-21T08:32:55.879Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1
level=info ts=2021-07-21T08:32:55.880Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1
level=info ts=2021-07-21T08:32:55.880Z caller=head.go:860 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=27.974µs wal_replay_duration=5.467247ms total_replay_duration=5.518124ms
level=info ts=2021-07-21T08:32:55.882Z caller=main.go:851 fs_type=EXT4_SUPER_MAGIC
level=info ts=2021-07-21T08:32:55.882Z caller=main.go:854 msg="TSDB started"
level=debug ts=2021-07-21T08:32:55.882Z caller=main.go:855 msg="TSDB options" MinBlockDuration=2h MaxBlockDuration=1d12h MaxBytes=0B NoLockfile=false RetentionDuration=15d WALSegmentSize=0B AllowOverlappingBlocks=false WALCompression=true
level=info ts=2021-07-21T08:32:55.882Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=debug ts=2021-07-21T08:32:55.882Z caller=manager.go:195 component="discovery manager scrape" msg="Starting provider" provider=static/0 subs=[prometheus]
level=info ts=2021-07-21T08:32:55.882Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=461.679µs remote_storage=1.813µs web_handler=351ns query_engine=852ns scrape=166.547µs scrape_sd=32.993µs notify=932ns notify_sd=2.044µs rules=1.303µs
level=info ts=2021-07-21T08:32:55.882Z caller=main.go:796 msg="Server is ready to receive web requests."
level=debug ts=2021-07-21T08:32:55.882Z caller=manager.go:213 component="discovery manager scrape" msg="Discoverer channel closed" provider=static/0
Operator logging
Duration: 5:00
Juju OLM provides capabilities to change the logging level for operators and interrogate operators logging output.
In the following steps, we will change the logging level for the current level and see how to interrogate the logs using Juju CLI.
The logging level of a model is defined in the model’s configuration options. We can display all of them by executing the following command:
$ juju model-config
Attribute From Value
agent-metadata-url default ""
agent-stream default released
agent-version model 2.9.10
apt-ftp-proxy default ""
apt-http-proxy default ""
apt-https-proxy default ""
apt-mirror default ""
apt-no-proxy default ""
automatically-retry-hooks default true
backup-dir default ""
charmhub-url default https://api.charmhub.io
cloudinit-userdata default ""
container-image-metadata-url default ""
container-image-stream default released
container-inherit-properties default ""
container-networking-method default ""
default-series default focal
default-space default ""
development default false
disable-network-management default false
egress-subnets default ""
enable-os-refresh-update default true
enable-os-upgrade default true
fan-config default ""
firewall-mode default instance
ftp-proxy default ""
http-proxy default ""
https-proxy default ""
ignore-machine-addresses default false
image-metadata-url default ""
image-stream default released
juju-ftp-proxy default ""
juju-http-proxy default ""
juju-https-proxy default ""
juju-no-proxy default 127.0.0.1,localhost,::1
logforward-enabled default false
logging-config model <root>=INFO
lxd-snap-channel default latest/stable
max-action-results-age default 336h
max-action-results-size default 5G
max-status-history-age default 336h
max-status-history-size default 5G
net-bond-reconfigure-delay default 17
no-proxy default 127.0.0.1,localhost,::1
operator-storage controller microk8s-hostpath
provisioner-harvest-mode default destroyed
proxy-ssh default false
resource-tags model {}
snap-http-proxy default ""
snap-https-proxy default ""
snap-store-assertions default ""
snap-store-proxy default ""
snap-store-proxy-url default ""
ssl-hostname-verification default true
storage-default-block-source model kubernetes
storage-default-filesystem-source model kubernetes
test-mode default false
transmit-vendor-metrics default true
update-status-hook-interval default 5m
workload-storage controller microk8s-hostpath
We can also display one specific option by passing its name as an argument to the command:
$ juju model-config logging-config
<root>=info
The logging-config option is what we are looking for. We can see this option in action by looking at the logging output from the Prometheus operator. Run the following command in a new terminal window, so you can watch the output:
$ juju debug-log --include prometheus-k8s
application-prometheus-k8s: 09:26:09 INFO juju.worker.caasoperator.uniter.prometheus-k8s/0.operation ran "config-changed" hook (via hook dispatching script: dispatch)
application-prometheus-k8s: 09:26:09 INFO juju.worker.caasoperator.uniter.prometheus-k8s/0 found queued "start" hook
application-prometheus-k8s: 09:26:09 INFO unit.prometheus-k8s/0.juju-log Running legacy hooks/start.
application-prometheus-k8s: 09:26:10 INFO juju.worker.caasoperator.uniter.prometheus-k8s/0.operation ran "start" hook (via hook dispatching script: dispatch)
application-prometheus-k8s: 09:26:12 INFO juju.worker.caasoperator.uniter.prometheus-k8s/0.operation ran "config-changed" hook (via hook dispatching script: dispatch)
application-prometheus-k8s: 09:26:18 INFO juju.worker.caasoperator started pod init on "prometheus-k8s/0"
application-prometheus-k8s: 09:26:19 INFO juju.worker.caasoperator.uniter.prometheus-k8s/0.operation ran "database-storage-attached" hook (via hook dispatching script: dispatch)
application-prometheus-k8s: 09:31:07 INFO juju.worker.caasoperator.uniter.prometheus-k8s/0.operation ran "update-status" hook (via hook dispatching script: dispatch)
application-prometheus-k8s: 09:32:44 INFO juju.worker.caasoperator.uniter.prometheus-k8s/0.operation ran "config-changed" hook (via hook dispatching script: dispatch)
application-prometheus-k8s: 09:32:52 INFO juju.worker.caasoperator started pod init on "prometheus-k8s/0"
As we can see all log messages are of the “INFO” level:
Now let’s change the value of the logging-config option to “DEBUG”.
$ juju model-config logging-config="<root>=debug"
Our log stream should now include more detail:
$ juju debug-log --include prometheus-k8s
application-prometheus-k8s: 09:26:10 INFO juju.worker.caasoperator.uniter.prometheus-k8s/0.operation ran "start" hook (via hook dispatching script: dispatch)
application-prometheus-k8s: 09:26:12 INFO juju.worker.caasoperator.uniter.prometheus-k8s/0.operation ran "config-changed" hook (via hook dispatching script: dispatch)
application-prometheus-k8s: 09:26:18 INFO juju.worker.caasoperator started pod init on "prometheus-k8s/0"
application-prometheus-k8s: 09:26:19 INFO juju.worker.caasoperator.uniter.prometheus-k8s/0.operation ran "database-storage-attached" hook (via hook dispatching script: dispatch)
application-prometheus-k8s: 09:31:07 INFO juju.worker.caasoperator.uniter.prometheus-k8s/0.operation ran "update-status" hook (via hook dispatching script: dispatch)
application-prometheus-k8s: 09:32:44 INFO juju.worker.caasoperator.uniter.prometheus-k8s/0.operation ran "config-changed" hook (via hook dispatching script: dispatch)
application-prometheus-k8s: 09:32:52 INFO juju.worker.caasoperator started pod init on "prometheus-k8s/0"
application-prometheus-k8s: 09:35:41 INFO juju.worker.caasoperator.uniter.prometheus-k8s/0.operation ran "update-status" hook (via hook dispatching script: dispatch)
application-prometheus-k8s: 09:35:57 DEBUG juju.worker.caasoperator.uniter.prometheus-k8s/0.remotestate got update status interval change for prometheus-k8s/0: ok=true
application-prometheus-k8s: 09:35:57 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.243,127.0.0.1,::1,controller-service.controller-mk8s.svc.cluster.local,localhost", AutoNoProxy:""}
Next steps
Duration: 2:00
Congratulations! You have reached the end of this tutorial.
You can now move to the next tutorial - “11. Debugging”.
In this tutorial you have learnt how to:
-
Interrogate logging output for both the application and operator workloads
-
View and change logging level for applications
-
View and change logging level for operators