Hi
I deployed charmed-Kubernetes with version 1.26/stable. After a successful deployment, I tried to upgrade the version from 1.26 to 1.27/stable. But upgrade failed. I need some help to fix the issue.
Command used to upgrade
juju refresh kubernetes-control-plane
juju config kubernetes-control-plane channel=1.27/stable
juju run kubernetes-control-plane/0 upgrade
juju run kubernetes-control-plane/1 upgrade
juju refresh kubernetes-worker
juju config kubernetes-worker channel=1.27/stable
juju run kubernetes-worker/0 upgrade
juju run kubernetes-worker/1 upgrade
Control-plane units are blocked with Stopped services: kubelet and worker units are blocked with Waiting for kubelet to start
Logs from kubernetes-control-plane/0
2024-02-03 11:15:44 INFO unit.kubernetes-control-plane/0.juju-log server.go:325 Invoking reactive handler: reactive/kubernetes_control_plane.py:3682:apply_node_labels
2024-02-03 11:15:45 INFO unit.kubernetes-control-plane/0.juju-log server.go:325 Invoking reactive handler: reactive/kubernetes_control_plane.py:3709:watch_dns_for_changes
2024-02-03 11:15:45 INFO unit.kubernetes-control-plane/0.juju-log server.go:325 Executing ['kubectl', '--kubeconfig=/root/.kube/config', 'get', 'service', '--namespace', 'kube-system', 'kube-dns', '--output', 'json']
2024-02-03 11:15:45 WARNING unit.kubernetes-control-plane/0.update-status logger.go:60 Failed to get unit file state for kube-apiserver.service: No such file or directory
2024-02-03 11:15:45 WARNING unit.kubernetes-control-plane/0.update-status logger.go:60 Failed to get unit file state for snap.kube-apiserver.service: No such file or directory
2024-02-03 11:15:45 INFO unit.kubernetes-control-plane/0.juju-log server.go:325 Checking if snap.kube-apiserver.daemon is active (0 / 6)
2024-02-03 11:15:45 WARNING unit.kubernetes-control-plane/0.update-status logger.go:60 Failed to get unit file state for kube-controller-manager.service: No such file or directory
2024-02-03 11:15:45 WARNING unit.kubernetes-control-plane/0.update-status logger.go:60 Failed to get unit file state for snap.kube-controller-manager.service: No such file or directory
2024-02-03 11:15:45 INFO unit.kubernetes-control-plane/0.juju-log server.go:325 Checking if snap.kube-controller-manager.daemon is active (0 / 6)
2024-02-03 11:15:45 WARNING unit.kubernetes-control-plane/0.update-status logger.go:60 Failed to get unit file state for kube-scheduler.service: No such file or directory
2024-02-03 11:15:45 WARNING unit.kubernetes-control-plane/0.update-status logger.go:60 Failed to get unit file state for snap.kube-scheduler.service: No such file or directory
2024-02-03 11:15:45 INFO unit.kubernetes-control-plane/0.juju-log server.go:325 Checking if snap.kube-scheduler.daemon is active (0 / 6)
2024-02-03 11:15:45 WARNING unit.kubernetes-control-plane/0.update-status logger.go:60 Failed to get unit file state for kube-proxy.service: No such file or directory
2024-02-03 11:15:45 WARNING unit.kubernetes-control-plane/0.update-status logger.go:60 Failed to get unit file state for snap.kube-proxy.service: No such file or directory
2024-02-03 11:15:45 INFO unit.kubernetes-control-plane/0.juju-log server.go:325 Checking if snap.kube-proxy.daemon is active (0 / 6)
2024-02-03 11:15:45 WARNING unit.kubernetes-control-plane/0.update-status logger.go:60 Failed to get unit file state for kubelet.service: No such file or directory
2024-02-03 11:15:45 WARNING unit.kubernetes-control-plane/0.update-status logger.go:60 Failed to get unit file state for snap.kubelet.service: No such file or directory
2024-02-03 11:15:45 INFO unit.kubernetes-control-plane/0.juju-log server.go:325 Checking if snap.kubelet.daemon is active (0 / 6)
2024-02-03 11:15:55 INFO unit.kubernetes-control-plane/0.juju-log server.go:325 Checking if snap.kubelet.daemon is active (1 / 6)
2024-02-03 11:16:05 INFO unit.kubernetes-control-plane/0.juju-log server.go:325 Checking if snap.kubelet.daemon is active (2 / 6)
2024-02-03 11:16:15 INFO unit.kubernetes-control-plane/0.juju-log server.go:325 Checking if snap.kubelet.daemon is active (3 / 6)
2024-02-03 11:16:16 WARNING unit.unit-kubernetes-control-plane-0.collect-metrics logger.go:60 /var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/hooks/collect-metrics:43: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
2024-02-03 11:16:16 WARNING unit.unit-kubernetes-control-plane-0.collect-metrics logger.go:60 doc = yaml.load(f)
2024-02-03 11:16:17 WARNING unit.unit-kubernetes-control-plane-0.collect-metrics logger.go:60 No resources found
2024-02-03 11:16:17 WARNING unit.unit-kubernetes-control-plane-0.collect-metrics logger.go:60 No resources found
2024-02-03 11:16:17 WARNING unit.unit-kubernetes-control-plane-0.collect-metrics logger.go:60 No resources found
2024-02-03 11:16:25 INFO unit.kubernetes-control-plane/0.juju-log server.go:325 Checking if snap.kubelet.daemon is active (4 / 6)
2024-02-03 11:16:35 INFO unit.kubernetes-control-plane/0.juju-log server.go:325 Checking if snap.kubelet.daemon is active (5 / 6)
2024-02-03 11:16:46 INFO unit.kubernetes-control-plane/0.juju-log server.go:325 status-set: blocked: Stopped services: kubelet
2024-02-03 11:16:46 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via explicit, bespoke hook script)
root@staging-maas:~# juju status
Model Controller Cloud/Region Version SLA Timestamp
k8s-staging maas-controller maas-one/default 3.1.7 unsupported 11:10:55Z
App Version Status Scale Charm Channel Rev Exposed Message
calico 3.21.4 active 4 calico 1.26/stable 77 no Calico is active
containerd 1.6.8 active 4 containerd 1.26/stable 54 no Container runtime available
easyrsa 3.0.1 active 1 easyrsa 1.26/stable 33 no Certificate Authority connected.
etcd 3.4.22 active 3 etcd 1.26/stable 728 no Healthy with 3 known peers
kubeapi-load-balancer 1.18.0 active 1 kubeapi-load-balancer 1.26/stable 76 yes Loadbalancer ready.
kubernetes-control-plane 1.27.9 blocked 2 kubernetes-control-plane 1.26/stable 247 no Stopped services: kubelet
kubernetes-worker 1.27.9 waiting 2 kubernetes-worker 1.26/stable 92 yes Waiting for kubelet to start.
Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0/lxd/0 10.90.2.80 Certificate Authority connected.
etcd/0* active idle 0/lxd/1 10.90.2.81 2379/tcp Healthy with 3 known peers
etcd/1 active idle 1/lxd/0 10.90.2.84 2379/tcp Healthy with 3 known peers
etcd/2 active idle 2/lxd/0 10.90.2.88 2379/tcp Healthy with 3 known peers
kubeapi-load-balancer/0* active idle 0/lxd/2 10.90.2.83 443,6443/tcp Loadbalancer ready.
kubernetes-control-plane/0* blocked idle 0/lxd/3 10.90.2.82 6443/tcp Stopped services: kubelet
calico/3 active idle 10.90.2.82 Calico is active
containerd/3 active idle 10.90.2.82 Container runtime available
kubernetes-control-plane/1 blocked idle 1/lxd/1 10.90.2.85 6443/tcp Stopped services: kubelet
calico/2 active idle 10.90.2.85 Calico is active
containerd/2 active idle 10.90.2.85 Container runtime available
kubernetes-worker/0 waiting idle 2/lxd/1 10.90.2.87 80,443/tcp Waiting for kubelet to start.
calico/0* active idle 10.90.2.87 Calico is active
containerd/0* active idle 10.90.2.87 Container runtime available
kubernetes-worker/1* waiting idle 3/lxd/0 10.90.2.86 80,443/tcp Waiting for kubelet to start.
calico/1 active idle 10.90.2.86 Calico is active
containerd/1 active idle 10.90.2.86 Container runtime available
Machine State Address Inst id Base AZ Message
0 started 10.90.2.76 controller-1 ubuntu@22.04 default Deployed
0/lxd/0 started 10.90.2.80 juju-8c949b-0-lxd-0 ubuntu@22.04 default Container started
0/lxd/1 started 10.90.2.81 juju-8c949b-0-lxd-1 ubuntu@22.04 default Container started
0/lxd/2 started 10.90.2.83 juju-8c949b-0-lxd-2 ubuntu@22.04 default Container started
0/lxd/3 started 10.90.2.82 juju-8c949b-0-lxd-3 ubuntu@22.04 default Container started
1 started 10.90.2.77 controller-2 ubuntu@22.04 default Deployed
1/lxd/0 started 10.90.2.84 juju-8c949b-1-lxd-0 ubuntu@22.04 default Container started
1/lxd/1 started 10.90.2.85 juju-8c949b-1-lxd-1 ubuntu@22.04 default Container started
2 started 10.90.2.78 worker-2 ubuntu@22.04 default Deployed
2/lxd/0 started 10.90.2.88 juju-8c949b-2-lxd-0 ubuntu@22.04 default Container started
2/lxd/1 started 10.90.2.87 juju-8c949b-2-lxd-1 ubuntu@22.04 default Container started
3 started 10.90.2.79 worker-1 ubuntu@22.04 default Deployed
3/lxd/0 started 10.90.2.86 juju-8c949b-3-lxd-0 ubuntu@22.04 default Container started