Hello everyone,
I tried to follow the tutorial on https://juju.is/docs/olm/get-started-with-juju and during the bootstrap, I got an error during controller pod creation - it stuck on CrashLoopBackOff.
Examining logs, I found that the path for secret “/var/lib/juju/server.pem” could not be found. Examining pod manifest, I got that the controller-server.pem has path set to template-server.pem instead of server.pem. I could not fix manually because when I forced recreation, the juju cli stops the bootstrap and performs cleanup.
I followed the steps below according to tutorial:
- Installed microk8s and enabled the required services
- Installed juju as described
- Performed
juju bootstrap microk8s tutorial-controller
I also performed the same command using --debug flag:
12:17:42 INFO juju.cmd supercommand.go:56 running juju [2.9.33 95186b2e0c2dfa9fe7b0b815cfe2c939813f9302 gc go1.18.3]
12:17:42 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/20276/bin/juju", "bootstrap", "microk8s", "tutorials", "--debug"}
12:17:43 DEBUG juju.kubernetes.provider provider.go:140 opening model "add-cloud".
12:17:43 DEBUG juju.kubernetes.provider metadata.go:169 use the default Storage class "microk8s-hostpath" for operator storage class because it also matches Juju preferred config {hostpath microk8s.io/hostpath map[] false WaitForFirstConsumer}
12:17:43 DEBUG juju.kubernetes.provider metadata.go:190 use the default Storage class "microk8s-hostpath" for nominated storage class
12:17:43 INFO cmd cloudcredential.go:47 updating credential store
12:17:43 DEBUG juju.cmd.juju.commands bootstrap.go:1307 authenticating with region "" and credential "microk8s" ()
12:17:43 DEBUG juju.cmd.juju.commands bootstrap.go:1455 provider attrs: map[operator-storage: workload-storage:]
12:17:44 INFO cmd authkeys.go:114 Adding contents of "/home/guilhermec/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
12:17:44 INFO cmd authkeys.go:114 Adding contents of "/home/guilhermec/.ssh/id_rsa.pub" to authorized-keys
12:17:44 DEBUG juju.cmd.juju.commands bootstrap.go:1530 preparing controller with config: map[agent-metadata-url: agent-stream:released apt-ftp-proxy: apt-http-proxy: apt-https-proxy: apt-mirror: apt-no-proxy: authorized-keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOmi4uTStHazogHVkgVi/pSfQZ2dnZKOtw78T4A6k0ic6XeDL1S7n7qGu++6DMHC75Creh7ZIz4PHxDsi6Unp0dcZY20FoGP8Jikccq/2PL3OeovVz6a58QTaPe1HKWjtSBRkvpb/lKzvuFiganua1Umj4kZozfyEBpJ60fgQP9hWXEy2ikkPb/eSiq5AjJXy6Ze1MeyQKvU0u2bOOh+s7MOkAmdGX3xBZn+uMJSna9BmWFsRkidhb/exX1u9ybDaGZljXBDZlHtI3QIZv13uFG3glFW2FpdWEakonmoQ1L6fgDU44SRmCU6YZJjjBG9bDKSsbJz2gnePx8I2cx9lD juju-client-key
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwYUATPfnE6KQP74c/3UdScSQI7Dx3HGkrw1Y0zxPENbkgyN+3xdbORy2Eor4fHo5R/OvPkFbXmdM7dV5BtQ4jIrxGnGBn9/QrGbeFGGmOn6txkqEf7cNNi0on9eUnlXESFFRCMPD/0Nigh5rWvChqYOgHAC5Rzkbqy1MFaYL1yrlWRL4q6SQnp9BUDphPAqHRchxi1zRRPEPJX1XsBUJ7xb9XYZr9AyWTcVCYTxfzbIM15cl29/zPSsRS5UZSdPcpGpOHfncPOEyLSEXDjxsVICzvvgSJDIOLDKs+kzL+z9QTjJwFzKz7jyASJDQBHGDqakeIoIIwlqbGVyQ8p6SrEQ68N3PgmFJh/ZXL2kthzcvA1tiAU7Y86wUg6VpDUKhfD9jltlW/DdmLi1z0YCE0thiV9FTPVdw0L++bBYBpsB+fkJEgyHAqIto9gAcy+RmP5WMT9V4dImpY5puQXPWxcIDxm1feyqlrHSls871XvUfRHZjJZQnqsAwJpRWf/Zk= guilhermec@WFMQ134
automatically-retry-hooks:true backup-dir: charmhub-url:https://api.charmhub.io cloudinit-userdata: container-image-metadata-url: container-image-stream:released container-inherit-properties: container-networking-method: default-series:focal default-space: development:false disable-network-management:false disable-telemetry:false egress-subnets: enable-os-refresh-update:true enable-os-upgrade:true fan-config: firewall-mode:instance ftp-proxy: http-proxy: https-proxy: ignore-machine-addresses:false image-metadata-url: image-stream:released juju-ftp-proxy: juju-http-proxy: juju-https-proxy: juju-no-proxy:127.0.0.1,localhost,::1 logforward-enabled:false logging-config: logging-output: lxd-snap-channel:latest/stable max-action-results-age:336h max-action-results-size:5G max-status-history-age:336h max-status-history-size:5G name:controller net-bond-reconfigure-delay:17 no-proxy:127.0.0.1,localhost,::1 num-container-provision-workers:4 num-provision-workers:16 operator-storage:microk8s-hostpath provisioner-harvest-mode:destroyed proxy-ssh:false resource-tags: snap-http-proxy: snap-https-proxy: snap-store-assertions: snap-store-proxy: snap-store-proxy-url: ssl-hostname-verification:true test-mode:false transmit-vendor-metrics:true type:kubernetes update-status-hook-interval:5m uuid:df6d73bc-32a7-4e1e-8a7c-92392ccf9edd workload-storage:microk8s-hostpath]
12:17:44 DEBUG juju.kubernetes.provider provider.go:140 opening model "controller".
12:17:44 INFO cmd bootstrap.go:855 Creating Juju controller "tutorials" on microk8s/localhost
12:17:44 INFO juju.cmd.juju.commands bootstrap.go:921 combined bootstrap constraints:
12:17:44 INFO cmd bootstrap.go:971 Bootstrap to Kubernetes cluster identified as microk8s/localhost
12:17:44 DEBUG juju.environs.simplestreams simplestreams.go:417 searching for signed metadata in datasource "gui simplestreams"
12:17:44 DEBUG juju.environs.simplestreams simplestreams.go:452 looking for data index using path streams/v1/index2.sjson
12:17:44 DEBUG juju.environs.simplestreams simplestreams.go:464 looking for data index using URL https://streams.canonical.com/juju/gui/streams/v1/index2.sjson
12:17:44 DEBUG juju.environs.simplestreams simplestreams.go:467 streams/v1/index2.sjson not accessed, actual error: [{github.com/juju/juju/environs/simplestreams.(*urlDataSource).Fetch:192: "https://streams.canonical.com/juju/gui/streams/v1/index2.sjson" not found}]
12:17:44 DEBUG juju.environs.simplestreams simplestreams.go:468 streams/v1/index2.sjson not accessed, trying legacy index path: streams/v1/index.sjson
12:17:45 DEBUG juju.environs.simplestreams simplestreams.go:487 read metadata index at "https://streams.canonical.com/juju/gui/streams/v1/index.sjson"
12:17:45 DEBUG juju.environs.simplestreams simplestreams.go:1019 finding products at path "streams/v1/com.canonical.streams-released-dashboard.sjson"
12:17:45 INFO cmd bootstrap.go:867 Fetching Juju Dashboard 0.8.1
12:17:45 DEBUG juju.kubernetes.provider k8s.go:470 controller pod config:
&{Tags:map[] Bootstrap:0xc0005fdc00 DisableSSLHostnameVerification:false ProxySettings:{Http: Https: Ftp: NoProxy:127.0.0.1,localhost,::1 AutoNoProxy:} Controller:map[agent-logfile-max-backups:2 agent-logfile-max-size:100M api-port:17070 api-port-open-delay:2s audit-log-capture-args:false audit-log-exclude-methods:[ReadOnlyMethods] audit-log-max-backups:10 audit-log-max-size:300M auditing-enabled:true batch-raft-fsm:false ca-cert:-----BEGIN CERTIFICATE-----
MIIEEzCCAnugAwIBAgIVANkJgBlW/Yc5aZJ81Hz42I+SkhJOMA0GCSqGSIb3DQEB
CwUAMCExDTALBgNVBAoTBEp1anUxEDAOBgNVBAMTB2p1anUtY2EwHhcNMjIwODMx
MTExMjQ0WhcNMzIwODMxMTExNzQ0WjAhMQ0wCwYDVQQKEwRKdWp1MRAwDgYDVQQD
EwdqdWp1LWNhMIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIBigKCAYEA2Ti6/UnM
QE5cUEIl8jzqKiNl/i2/VoBkwsTUbEWNreHDhyTviar1Co5gEqJ1VOgO4A+FqUq3
WsAL9kzBncYWcQZ6ubA0Xpj30DFX2lYIcV4pn4nGHuxi9PUttvqT4cB9DlGXVShR
2Y4xyfXsoq73PVQk16x1T8LXfwb8HpryB3IhooOH3jte3s8DjjmqGgKa/kzwCZxN
/ENQN/BvKMi3NtAYKx/qoalofewCXK2tUjudPnMYNMRZGAu5jHgZJLK6L20Z0kkN
1uEJTQx7yKoReQkNkwlZESTEhHaYKiqDn1DkR5KNIoAvThhPip3DqyHS3LeiJDwN
8U232MptSOhoBz976HIxACv5M5ab7j07slYwzGyYDyyWxlb8QAekvH/TOA5eyB7c
VC5ssixY/OUy4o4CClXKXpT3uqVbHuAUsMc3YWtghjgQB60pjQBqT3310C9oPKU9
fpjwQfaKYwrK0/S6AAmdzYpGBTje2Idnp1znrm++jRbvoPp6RndOCB6ZAgMBAAGj
QjBAMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTc
4d3sihaB3S/lHi+l4JVEaYpRQTANBgkqhkiG9w0BAQsFAAOCAYEA0rNwuENxbd37
BhR3cc2e495Lx4flu03jBOzBY5U1VM1eIhYR1zRzZqn0ON5cBEK+skyWCQbDabsE
PkxmziKNhz0FuXSAwWmVAA9mWOkcP7IWBKASwEd80tLMYlmYobGlwZbIFpabdyH3
mF6Yj242bGjOEaR8ybXsEZTbNDoPYA+my5sfQ0JiI2QEf0NikK+i0wpfRXsOAIv1
HtwE+lsFUD1NNYMGu1/OM/42Ds0QhXnXSt9+8xFBzYEdy2SKV44QBBwdKTQnSc7X
VCPsxLz0d5ep5goZP44FS4DIdu/KxfUhDjoBakw9LiVRN9W9nebGTLLjOgL+1pCK
iWPCGpZ88k/t3h1mGuvCmMbMqGBDoTNVl3FNahunYn744GGbEj8d83QqG1m4pe3u
whXOtu+zOzAOlb1DK0CDNSyR53FvFMvqdjdO5WYISF3bwOrdTiE6dTA2FuA34Wp8
QWw54H3LQl7WucNbVzri6yfDPHwTOJ7t3/zec+knWPw9HVduetJc
-----END CERTIFICATE-----
charmstore-url:https://api.jujucharms.com/charmstore controller-name:tutorials controller-uuid:e6ce2b6d-cb7e-4403-8b0b-d79659777bd4 juju-db-snap-channel:4.4/stable max-agent-state-size:524288 max-charm-state-size:2097152 max-debug-log-duration:24h0m0s max-prune-txn-batch-size:1000000 max-prune-txn-passes:100 max-txn-log-size:10M metering-url:https://api.jujucharms.com/omnibus/v3 migration-agent-wait-time:15m model-logfile-max-backups:2 model-logfile-max-size:10M model-logs-size:20M mongo-memory-profile:default non-synced-writes-to-raft-log:false prune-txn-query-count:1000 prune-txn-sleep-time:10ms set-numa-control-policy:false state-port:37017] APIInfo:0xc000229b00 ControllerTag:controller-e6ce2b6d-cb7e-4403-8b0b-d79659777bd4 ControllerName:tutorials JujuVersion:2.9.33 DataDir:/var/lib/juju LogDir:/var/log/juju MetricsSpoolDir:/var/lib/juju/metricspool ControllerId:0 AgentEnvironment:map[PROVIDER_TYPE:kubernetes]}
12:17:45 INFO cmd bootstrap.go:394 Creating k8s resources for controller "controller-tutorials"
12:17:45 DEBUG juju.kubernetes.provider bootstrap.go:627 creating controller service:
&Service{ObjectMeta:{controller-service controller-tutorials 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/managed-by:juju app.kubernetes.io/name:controller] map[controller.juju.is/id:e6ce2b6d-cb7e-4403-8b0b-d79659777bd4] [] [] []},Spec:ServiceSpec{Ports:[]ServicePort{ServicePort{Name:api-server,Protocol:,Port:17070,TargetPort:{0 17070 },NodePort:0,AppProtocol:nil,},},Selector:map[string]string{app.kubernetes.io/name: controller,},ClusterIP:,Type:ClusterIP,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,IPFamilyPolicy:nil,ClusterIPs:[],IPFamilies:[],AllocateLoadBalancerNodePorts:nil,LoadBalancerClass:nil,InternalTrafficPolicy:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[]LoadBalancerIngress{},},Conditions:[]Condition{},},}
12:17:45 DEBUG juju.caas.kubernetes.provider.proxy setup.go:179 polling caas credential rbac secret, in 1 attempt, token for secret "controller-proxy" not found
12:17:47 DEBUG juju.kubernetes.provider configmap.go:84 updating configmap "controller-configmap"
12:17:48 DEBUG juju.kubernetes.provider configmap.go:84 updating configmap "controller-configmap"
12:17:49 DEBUG juju.kubernetes.provider bootstrap.go:1207 mongodb container args:
printf 'args="--dbpath=/var/lib/juju/db --sslPEMKeyFile=/var/lib/juju/server.pem --sslPEMKeyPassword=ignored --sslMode=requireSSL --port=37017 --journal --replSet=juju --quiet --oplogSize=1024 --auth --keyFile=/var/lib/juju/shared-secret --storageEngine=wiredTiger --bind_ip_all"\nipv6Disabled=$(sysctl net.ipv6.conf.all.disable_ipv6 -n)\nif [ $ipv6Disabled -eq 0 ]; then\n args="${args} --ipv6"\nfi\nexec mongod ${args}\n'>/root/mongo.sh && chmod a+x /root/mongo.sh && /root/mongo.sh
12:17:49 DEBUG juju.kubernetes.provider k8s.go:2252 selecting units "app.kubernetes.io/name=controller" to watch
12:17:49 DEBUG juju.kubernetes.provider.watcher k8swatcher.go:114 fire notify watcher for controller-0
12:17:49 DEBUG juju.kubernetes.provider.watcher k8swatcher.go:114 fire notify watcher for controller
12:17:56 DEBUG juju.kubernetes.provider.watcher k8swatcher.go:114 fire notify watcher for controller
12:17:56 DEBUG juju.kubernetes.provider bootstrap.go:957 Successfully assigned controller-tutorials/controller-0 to wfmq134
12:17:56 DEBUG juju.kubernetes.provider bootstrap.go:957 Pulled images
12:17:56 DEBUG juju.kubernetes.provider bootstrap.go:957 Created container mongodb
12:17:56 DEBUG juju.kubernetes.provider bootstrap.go:957 Started mongodb container
12:17:56 DEBUG juju.kubernetes.provider bootstrap.go:957 Created container api-server
12:17:56 DEBUG juju.kubernetes.provider.watcher k8swatcher.go:114 fire notify watcher for controller-0
12:17:57 DEBUG juju.kubernetes.provider.watcher k8swatcher.go:114 fire notify watcher for controller-0
12:17:57 DEBUG juju.kubernetes.provider bootstrap.go:957 Started controller container
12:17:58 DEBUG juju.kubernetes.provider.watcher k8swatcher.go:114 fire notify watcher for controller
12:17:58 INFO cmd bootstrap.go:1045 Starting controller pod
12:17:58 INFO cmd bootstrap.go:711 Bootstrap agent now started
12:17:58 INFO juju.juju api.go:330 API endpoints changed from [] to [10.152.183.6:17070]
12:17:58 INFO cmd controller.go:88 Contacting Juju controller at 10.152.183.6 to verify accessibility...
12:17:58 INFO juju.juju api.go:78 connecting to API addresses: [10.152.183.6:17070]
12:27:58 INFO cmd controller.go:141 Still waiting for API to become available: starting proxy for api connection: connecting k8s proxy: waiting for pod controller-0 to become ready for tunnel: context deadline exceeded
12:28:01 INFO juju.juju api.go:78 connecting to API addresses: [10.152.183.6:17070]
Microk8s pod status:
(base) guilhermec@WFMQ134:~$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-857f7b9f7d-w8rhz 1/1 Running 4 (3h11m ago) 2d1h
kube-system coredns-66bcf65bb8-lmffq 1/1 Running 4 (3h11m ago) 2d1h
kube-system calico-node-dzkrr 1/1 Running 4 (3h11m ago) 2d1h
ingress nginx-ingress-microk8s-controller-rlpcz 1/1 Running 0 85m
kube-system hostpath-provisioner-f57964d5f-p6b68 1/1 Running 0 83m
controller-tutorials controller-0 1/2 CrashLoopBackOff 7 (3m28s ago) 14m
Container logs:
(base) guilhermec@WFMQ134:~$ kubectl logs controller-0 --namespace controller-tutorials api-server
Installing Dashboard...
curl: (6) Could not resolve host: streams.canonical.com
curl: (6) Could not resolve host: streams.canonical.com
curl: (6) Could not resolve host: streams.canonical.com
curl: (6) Could not resolve host: streams.canonical.com
curl: (6) Could not resolve host: streams.canonical.com
curl: (6) Could not resolve host: streams.canonical.com
curl: (6) Could not resolve host: streams.canonical.com
curl: (6) Could not resolve host: streams.canonical.com
curl: (6) Could not resolve host: streams.canonical.com
curl: (6) Could not resolve host: streams.canonical.com
(base) guilhermec@WFMQ134:~$ kubectl logs controller-0 --namespace controller-tutorials mongodb
{"t":{"$date":"2022-08-31T11:29:10.437Z"},"s":"W", "c":"CONTROL", "id":23322, "ctx":"main","msg":"Option: sslMode is deprecated. Please use tlsMode instead."}
{"t":{"$date":"2022-08-31T11:29:10.437Z"},"s":"W", "c":"CONTROL", "id":23321, "ctx":"main","msg":"Option: This name is deprecated. Please use the preferred name instead.","attr":{"deprecatedName":"sslPEMKeyFile","preferredName":"tlsCertificateKeyFile"}}
{"t":{"$date":"2022-08-31T11:29:10.437Z"},"s":"W", "c":"CONTROL", "id":23321, "ctx":"main","msg":"Option: This name is deprecated. Please use the preferred name instead.","attr":{"deprecatedName":"sslPEMKeyPassword","preferredName":"tlsCertificateKeyFilePassword"}}
{"t":{"$date":"2022-08-31T11:29:10.439+00:00"},"s":"E", "c":"NETWORK", "id":23248, "ctx":"main","msg":"Cannot read certificate file","attr":{"keyFile":"/var/lib/juju/server.pem","error":"error:02001002:system library:fopen:No such file or directory"}}
{"t":{"$date":"2022-08-31T11:29:10.439+00:00"},"s":"F", "c":"CONTROL", "id":20574, "ctx":"main","msg":"Error during global initialization","attr":{"error":{"code":140,"codeName":"InvalidSSLConfiguration","errmsg":"Can not set up PEM key file."}}}
Secret on manifest:
controller-server-pem
secret:
defaultMode: 256
items:
- key: server.pem
path: template-server.pem
Volume mounts:
- mountPath: /var/lib/juju/template-server.pem
name: controller-server-pem
readOnly: true
subPath: template-server.pem
Mongo container definition:
- args:
- -c
- printf 'args="--dbpath=/var/lib/juju/db --sslPEMKeyFile=/var/lib/juju/server.pem
--sslPEMKeyPassword=ignored --sslMode=requireSSL --port=37017 --journal --replSet=juju
--quiet --oplogSize=1024 --auth --keyFile=/var/lib/juju/shared-secret --storageEngine=wiredTiger
--bind_ip_all"\nipv6Disabled=$(sysctl net.ipv6.conf.all.disable_ipv6 -n)\nif
[ $ipv6Disabled -eq 0 ]; then\n args="${args} --ipv6"\nfi\nexec mongod ${args}\n'>/root/mongo.sh
&& chmod a+x /root/mongo.sh && /root/mongo.sh
command:
- /bin/sh
image: jujusolutions/juju-db:4.4
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- mongo
- --port=37017
- --ssl
- --sslAllowInvalidHostnames
- --sslAllowInvalidCertificates
- --sslPEMKeyFile=/var/lib/juju/server.pem
- --eval
- db.adminCommand('ping')
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: mongodb
If anyone can help, I would appreciate.
Thanks in advance.