I installed charmed-kubernetes in Openstack and ran in to a problem with the OCCM pods installed by openstack-integrator charm (the cdk-addons module). The OCCM pods are crashing on startup with the message
runtime: mlock of signal stack failed: 12
runtime: increase the mlock limit (ulimit -l) or
runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+
fatal error: mlock failed
I think this is caused by LimitMEMLOCK in the juju .service files.
Many thanks, this is interesting. Are you able to explain the following please?
routergod@management:~$ juju status kubernetes-worker/0
Model Controller Cloud/Region Version SLA Timestamp
k8s domaintrust openstack/RegionOne 2.8.3 unsupported 18:44:39Z
App Version Status Scale Charm Store Rev OS Notes
containerd 1.3.3 active 1 containerd jujucharms 94 ubuntu
flannel 0.11.0 active 1 flannel jujucharms 506 ubuntu
kubernetes-worker 1.19.0 active 1 kubernetes-worker jujucharms 704 ubuntu exposed
Unit Workload Agent Machine Public address Ports Message
kubernetes-worker/0* active idle 6 10.0.27.186 80/tcp,443/tcp Kubernetes worker running.
containerd/1 active idle 10.0.27.186 Container runtime available
flannel/1 active idle 10.0.27.186 Flannel subnet 10.1.30.1/24
Machine State DNS Inst id Series AZ Message
6 started 10.0.27.186 d3e5f404-704b-42e0-9a8a-d2759e182817 focal nova ACTIVE
routergod@management:~$ juju run --machine 6 -- ulimit -l
65536
routergod@management:~$ juju run --unit kubernetes-worker/0 -- ulimit -l
64
I canāt explain this at the moment. I tried it for a simple test deployment and got 64 in both cases.
Just to rule out something weird, you could juju run ip address or something to confirm that the same target machine is being hit in both cases.
If you ssh into the machine and run ulimit directly, I wonder which of the two scenarios it matches. Thereās definitely something weird happening that I have no answer for.
I wonder if this is a case of something about the process/charm itself configuring ulimits. On LXD I definitely see 64 in both cases.
Even with ssh, I get:
As for LimitMEMLOCK being set, I know the systemd wrapper we have has support for changing memlock (and we set memlock to UNLIMITED for juju-db).
I donāt see other locations where juju is trying to control those things for unit agents, etc.
I also wonder if those are bionic vs focal vs xenial differences.
To track down this issue, start backtracking the process tree to see where values are changed.
Use pstree to see the process tree.
pstree
Use then cat /proc/<PID>/limits
To see if you have a unwanted setting and from that discover where you lose your setting. The parent process will normally be setting the child limits.
So this is a Focal systemd thing, not a juju thing and seems like the normal behavior. The values here line up with the LimitMEMLOCK settings for the services, although these are all inherited. There is no .service file in /usr/lib/systemd/*/ which sets LimitMEMLOCK explicitly.
When I log in interactively I get the higher limit;
In my case is appears that the 64K limit not sufficient to allow the openstack-cloud-controller-manager pods to execute. I manually added LimitMEMLOCK=infinity to the containerd.service files and now my OCCM pods are running.
Thing is I donāt get that the higher limit is actually required for the OCCM. And clearly this is not specifically a juju problem that I am suffering. I already reported this as a CDK Addons bug. To be honest I donāt really know where it sits.
All that aside, it seems that ārun --machineā is reporting the systemd value, not the jujud-machine-x value, for some reason?