Reserving kubenetes and system memory does not decrease allocatable memory amount

Not sure if I am checking it right, but reserving resources via system-reserved and kube-reserved, should decrease available resources. At least according to documentation it should be that away. Also additional internet resources shows that it should be the case.

However on Juju deployed environment I do not see this. Is it a bug or I am configuring/checking something wrong way? I have added reservations due to workload tries to allocate more memory, than available on node and Linux OOM kills too much processes and node becomes unresponsive. I need to reboot the VM to recover node.

Environment info:
juju: 2.7.4
kubernetes-worker 1.15.12

admin @ jujuhost:~ $ kubectl describe nodes juju-faa2ed-12 | grep -E "Allocatable:|Capacity:" -A 5rese
Capacity:
  cpu:                6
  ephemeral-storage:  64860696Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             16425280Ki
--
Allocatable:
  cpu:                6
  ephemeral-storage:  59775617335
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             16322880Ki
admin @ jujuhost:~ $
admin @ jujuhost:~ $
admin @ jujuhost:~ $ juju config kubernetes-worker kubelet-extra-config='{"system-reserved": {"cpu": "500m","memory": "64Mi"}, "kube-reserved": {"cpu": "500m","memory": "64Mi"}}'
admin @ jujuhost:~ $ kubectl describe nodes juju-faa2ed-12 | grep -E "Allocatable:|Capacity:" -A 5
Capacity:
  cpu:                6
  ephemeral-storage:  64860696Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             16425280Ki
--
Allocatable:
  cpu:                6
  ephemeral-storage:  59775617335
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             16322880Ki
admin @ jujuhost:~ $ juju config kubernetes-worker kubelet-extra-config='{"system-reserved": {"cpu": "500m","memory": "512Mi"}, "kube-reserved": {"cpu": "500m","memory": "512Mi"}}'
admin @ jujuhost:~ $ kubectl describe nodes juju-faa2ed-12 | grep -E "Allocatable:|Capacity:" -A 5
Capacity:
  cpu:                6
  ephemeral-storage:  64860696Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             16425280Ki
--
Allocatable:
  cpu:                6
  ephemeral-storage:  59775617335
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             16322880Ki
admin @ jujuhost:~ $

Can anyone from @k8s-charmers provide any advice? Any recommendations for verifying that the extra config is being correctly sent to kubelet?

I see these options applied to /root/cdk/kubelet/config.yaml file on a worker node. This file is presumably used for kubelet configuration. At least I guess that by looking at ps -ef output.

Ok it seems reason is wrong names for the configuration as CLI option names does not fit to config file options. I have verified that with streamingConnectionIdleTimeout/streaming-connection-idle-timeout option. Camelcase option name works as expected.

Now question would be, what are right option names? :slight_smile:

This command ends up in non ready nodes:
juju config kubernetes-worker kubelet-extra-config='{"featureGates": {"VolumeSnapshotDataSource": "true"},"systemReserved": {"cpu": "500m","memory": "512Mi "}, "kubeReserved": {"cpu": "500m","memory": "256Mi"}, "streamingConnectionIdleTimeout": "3h"}'

UPDATE: It appears option “featureGates”: {“VolumeSnapshotDataSource”: “true”} is not correct. Removing it, properly applies memory reservations. So case can be closed. Although Kubernetes documentation certainly should be clearer, when they are redirecting to source code for the deprecated options. :slight_smile: