As title, metacontroller-operator-0 pod is stuck in CrashLoopBackOff. Wondering if anyone else experienced this issue as well or know how to fix it.
Deployed Kubeflow on a microk8s cluster following this guide: How to install Kubeflow v1.6 | Documentation | Charmed Kubeflow.
Logs from the pod:
2022-10-13T20:27:29.404Z [pebble] HTTP API server listening on ":38812".
2022-10-13T20:27:29.404Z [pebble] Started daemon.
2022-10-13T20:27:29.443Z [pebble] POST /v1/services 38.631669ms 202
2022-10-13T20:27:29.443Z [pebble] Started default services with change 53.
2022-10-13T20:27:29.462Z [pebble] Service "container-agent" starting: /charm/bin/containeragent unit --data-dir /var/lib/juju --append-env "PATH=$PATH:/charm/bin" --show-log --charm-modified-version 0
2022-10-13T20:27:29.485Z [container-agent] 2022-10-13 20:27:29 INFO juju.cmd supercommand.go:56 running containerAgent [2.9.35 da3416008ea4ce7851a4c967ae191a0044917024 gc go1.19.2]
2022-10-13T20:27:29.485Z [container-agent] starting containeragent unit command
2022-10-13T20:27:29.485Z [container-agent] containeragent unit "unit-metacontroller-operator-0" start (2.9.35 [gc])
2022-10-13T20:27:29.485Z [container-agent] 2022-10-13 20:27:29 INFO juju.cmd.containeragent.unit runner.go:556 start "unit"
2022-10-13T20:27:29.485Z [container-agent] 2022-10-13 20:27:29 INFO juju.worker.upgradesteps worker.go:60 upgrade steps for 2.9.35 have already been run.
2022-10-13T20:27:29.486Z [container-agent] 2022-10-13 20:27:29 INFO juju.worker.probehttpserver server.go:157 starting http server on [::]:65301
2022-10-13T20:27:29.494Z [container-agent] 2022-10-13 20:27:29 INFO juju.api apiclient.go:1055 cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: operation was canceled
2022-10-13T20:27:29.494Z [container-agent] 2022-10-13 20:27:29 INFO juju.api apiclient.go:688 connection established to "wss://10.152.183.28:17070/model/12ffc8a9-4d76-414b-8a49-7e7bdc06cf00/api"
2022-10-13T20:27:29.497Z [container-agent] 2022-10-13 20:27:29 INFO juju.worker.apicaller connect.go:163 [12ffc8] "unit-metacontroller-operator-0" successfully connected to "10.152.183.28:17070"
2022-10-13T20:27:29.513Z [container-agent] 2022-10-13 20:27:29 INFO juju.api apiclient.go:1055 cannot resolve "controller-service.controller-my-controller.svc.cluster.local": lookup controller-service.controller-my-controller.svc.cluster.local: operation was canceled
2022-10-13T20:27:29.513Z [container-agent] 2022-10-13 20:27:29 INFO juju.api apiclient.go:688 connection established to "wss://10.152.183.28:17070/model/12ffc8a9-4d76-414b-8a49-7e7bdc06cf00/api"
2022-10-13T20:27:29.516Z [container-agent] 2022-10-13 20:27:29 INFO juju.worker.apicaller connect.go:163 [12ffc8] "unit-metacontroller-operator-0" successfully connected to "10.152.183.28:17070"
2022-10-13T20:27:29.537Z [container-agent] 2022-10-13 20:27:29 INFO juju.worker.migrationminion worker.go:142 migration phase is now: NONE
2022-10-13T20:27:29.538Z [container-agent] 2022-10-13 20:27:29 INFO juju.worker.logger logger.go:120 logger worker started
2022-10-13T20:27:29.541Z [container-agent] 2022-10-13 20:27:29 WARNING juju.worker.proxyupdater proxyupdater.go:282 unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
2022-10-13T20:27:29.552Z [container-agent] 2022-10-13 20:27:29 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-metacontroller-operator-0
2022-10-13T20:27:29.560Z [container-agent] 2022-10-13 20:27:29 INFO juju.worker.uniter uniter.go:326 unit "metacontroller-operator/0" started
2022-10-13T20:27:29.561Z [container-agent] 2022-10-13 20:27:29 INFO juju.worker.uniter uniter.go:631 resuming charm install
2022-10-13T20:27:29.576Z [container-agent] 2022-10-13 20:27:29 INFO juju.worker.caasupgrader upgrader.go:113 abort check blocked until version event received
2022-10-13T20:27:29.576Z [container-agent] 2022-10-13 20:27:29 INFO juju.worker.caasupgrader upgrader.go:119 unblocking abort check
2022-10-13T20:27:29.600Z [container-agent] 2022-10-13 20:27:29 INFO juju.worker.leadership tracker.go:194 metacontroller-operator/0 promoted to leadership of metacontroller-operator
2022-10-13T20:27:29.603Z [container-agent] 2022-10-13 20:27:29 INFO juju.worker.uniter.charm manifest_deployer.go:182 detected interrupted deploy of charm "ch:amd64/focal/metacontroller-operator-2"
2022-10-13T20:27:39.426Z [pebble] Check "readiness" failure 1 (threshold 3): received non-20x status code 418
2022-10-13T20:27:49.429Z [pebble] Check "readiness" failure 2 (threshold 3): received non-20x status code 418
2022-10-13T20:27:59.432Z [pebble] Check "readiness" failure 3 (threshold 3): received non-20x status code 418
2022-10-13T20:27:59.432Z [pebble] Check "readiness" failure threshold 3 hit, triggering action
2022-10-13T20:27:59.432Z [pebble] Service "container-agent" on-check-failure action is "shutdown", triggering server exit
2022-10-13T20:27:59.432Z [pebble] Server exiting!
2022-10-13T20:27:59.553Z [pebble] Stopping all running services.
2022-10-13T20:27:59.622Z [container-agent] 2022-10-13 20:27:59 INFO juju.worker.caasunitterminationworker worker.go:82 terminating due to SIGTERM
2022-10-13T20:28:04.628Z [pebble] Service "container-agent" stopped
Any help is appreciated. Thanks.