Juju worker (k8s): disk usage analysis problem?

juju
,
#1

Hi all,

we are investigating disk usage in our juju k8s cluster and we found this folder in our juju worker node:

/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots

this folder uses over 100GB of disk space (our juju cluster is running Gitlab runner).
We found over 10k folders there created over 2-3 months ago.
What is this folder for ? And is there is any automatic way to purge it ?

#2

Any ideas ? Hints ?

#3

Hi

This doesn’t look like it’s strictly a Juju concern, in that containerd is the container runtime that your underlying Kubernetes cloud is using. That snapshots directory is where containerd caches the various layers of the OCI images that your Kubernetes cluster deploys.

From memory, you could use crictl to clear these up provided that crictl is configured to talk to the correct containerd instance - something like crictl rmi --prune would remove unused cached layers.

My understanding is that there should be some automatic pruning as disk usage grows, but containerd will be unable to prune where layers are in use by running containers.

Which Kubernetes is this running on?

Cheers, Jon

#4

This is Juju k8s (1.18) installed on-premises - that is why I asked here :slight_smile:

#5

So this is Charmed Kubernetes?

I think the above still applies, but @kwmonroe or @addyess might be able to offer more specific guidance :slight_smile:

#6

Yes, it is :slight_smile:

#7

Have you tried a docker prune?