PVC Write permissions

Me again,

If I deploy Grafana or Prometheus into my Cinder backed OVH Kubernetes cluster, the mounted filesystems get a permission denied error.

Pretty common for both it seems judging by the google and the course of action seems to be:

https://stackoverflow.com/questions/60727107/how-can-i-give-grafana-user-appropriate-permission-so-that-it-can-start-successf

Setting

      securityContext:
          runAsUser: 472
          fsGroup: 472

I don’t see either in the securityGroup in the Kubernetes spec 3 tests or elsewhere. Can I set that, or is there an alternative course of action?

I think this might work (I’ve not tested it):

version: 3
containers:
  - name: something
    image: something/latest
    ports:
      - containerPort: 80
        name: fred
    kubernetes:
      securityContext:
        runAsUser: 472
        fsGroup: 472

Yeah I tried this in v2 and v3 spec and I get an error:

 config_content = self._build_grafana_ini()
        spec = {
            "version":3,
            "containers": [
                {
                    "name": self.app.name,
                    "imageDetails": {"imagePath": "grafana/grafana:7.2.1-ubuntu"},
                    "ports": [{"containerPort": port, "protocol": "TCP"}],
                    "volumeConfig": {
                          "name": "grafana-config-ini",
                            "mountPath": "/etc/grafana",
                            "files": {"grafana.ini": config_content},
                    },
                    "kubernetes": {
                    "securityContext":{
                      #"runAsNonRoot": true
                      "runAsUser": 472,
                      "fsGroup": 472,
                    },
                    "readinessProbe": {
                        "httpGet": {"path": "/api/health", "port": port},
                        "initialDelaySeconds": 10,
                        "timeoutSeconds": 30,
                    },
                  },
                }
            ]
        }
ops.model.ModelError: b'ERROR json: unknown field "fsGroup"\n'

Which suggests to me that the securityContext tag is in the correct place but it hasn’t got a clue what to do with the fsGroup value.

Ah sorry, there’s a difference between container security context and pod security context.
This should work.

version: 3
containers:
  - name: something
    image: something/latest
    ports:
      - containerPort: 80
        name: web
kubernetesResources:
  pod:
    securityContext:
      runAsUser: 32
      fsGroup: 32

Sorry @wallyworld still not quite sussed this.

Here’s my spec:

        spec = {
            "version": 3,
            "kubernetesResources":{
                "pod":{
                    "securityContext":{
                        "fsGroup": 1001,
                        "runAsUser": 1001,
                        "runAsGroup":1001,
                    }
                }
            },
            "containers": [
                {
                    "name": self.app.name,
                    "imageDetails": image_details,
                    "imagePullPolicy": "Always",
                    "ports": ports,
                },
            ],
        } 

it validated and deployed but still fails. And if I edit the podspec on the cluster it says:

 containers:
  - image: bitnami/solr:8.7.0-debian-10-r31
    imagePullPolicy: Always
    name: solr
    ports:
    - containerPort: 8983
      name: solr
      protocol: TCP
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      readOnlyRootFilesystem: false
      runAsNonRoot: false

Which clearly doesn’t have the stuff I specified in it.

I just deployed a small charm with the fsGroup set and via some debugging confirmed that the values are set in the statefulset Spec.Template when the statefulset create API is called.

We’ll have to do some investigation as to why aspects of the Spec.Template are not being applied as requested.

{
...
  "containers": [
    {
      "name": "mariadb-k8s",
      "image": "mariadb",
      "ports": [
        {
          "containerPort": 3306,
          "protocol": "TCP"
        }
      ],
      "env": [
        {
          "name": "MYSQL_DATABASE",
          "value": "database"
        },
        {
          "name": "MYSQL_PASSWORD",
          "value": "password"
        },
        {
          "name": "MYSQL_ROOT_PASSWORD",
          "value": "root"
        },
        {
          "name": "MYSQL_USER",
          "value": "admin"
        },
        {
          "name": "NODE_NAME",
          "valueFrom": {
            "fieldRef": {
              "apiVersion": "v1",
              "fieldPath": "spec.nodeName"
            }
          }
        }
      ],
      "resources": {},
      "volumeMounts": [
        {
          "name": "juju-data-dir",
          "mountPath": "/var/lib/juju"
        },
        {
          "name": "juju-data-dir",
          "mountPath": "/usr/bin/juju-run",
          "subPath": "tools/jujud"
        },
        {
          "name": "mariadb-k8s-configurations-config",
          "mountPath": "/etc/mysql/conf.d"
        }
      ],
      "securityContext": {
        "runAsNonRoot": false,
        "readOnlyRootFilesystem": false,
        "allowPrivilegeEscalation": true
      }
    }
  ],
  "serviceAccountName": "mariadb-k8s",
  "automountServiceAccountToken": true,
  "securityContext": {
    "fsGroup": 2
  }
}

Hmm fair enough & thanks for checking. The other thing I see people do is setup an init container and chown the mount directory space prior to the main container coming up. I tried that also earlier but it appears the volumes specified in metadata.yaml aren’t mounted in init pods, is there anyway to get that pvc mounted in the init pod?

Currently any Juju storage pv is only mounted in the workload container.
We could look at if it makes sense to also mount them in an init container to allow the sort of initialisation you are talking about. But then there’s issues reclaim policy to consider etc. I’d also like to understand why the fsGroup is not being applied.

Yeah,

I don’t think either of these methods are particularly weird for non-root running containers, but the setting of fsGroup and the runAsUser etc seem to be the correct way of going about it. chowning a bunch of stuff in an init container is particularly unelegant.

This also obviously affects any non-root charm in the charmstore, I saw it with, I believe grafana as well the other day when testing, so it’ll be cool to get this figured out and stuck into various K8S charms that currently mount volumes with the wrong permissions.

So did a bit more digging @wallyworld

I deployed my charm on microk8s and it worked without modification, which is what I was expecting as ya’ll deploy charms with pvc’s without bother.

So I then edited the statefulset and realised there are 2 securityContexts, and this setting only seems to stick in the one outside of the containers: block at spec level.

As soon as I added:

      securityContext:
        fsGroup: 1001

To the outer context, the pod came to life and fixed the permissions error I was facing.

So, can you set higher level kubernetes resources?

Thanks

Tom

You are referring to the PodSecurityContext value on the statefulset spec template right?

type StatefulSetSpec struct {
	Template v1.PodTemplateSpec
}

type PodTemplateSpec struct {
	metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
	Spec PodSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
}

type PodSpec struct {
	Volumes []Volume `json:"volumes,omitempty" patchStrategy:"merge,retainKeys" patchMergeKey:"name" protobuf:"bytes,1,rep,name=volumes"`
	InitContainers []Container `json:"initContainers,omitempty" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,20,rep,name=initContainers"`
	Containers []Container `json:"containers" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,2,rep,name=containers"`
	// SecurityContext holds pod-level security attributes and common container settings.
	// Optional: Defaults to empty.  See type description for default values of each field.
	// +optional
	SecurityContext *PodSecurityContext `json:"securityContext,omitempty" protobuf:"bytes,14,opt,name=securityContext"`
}

The SecurityContext in the PodSpec above is what the securityContext passed to Juju is set from

version: 3
containers:
  - name: something
    image: something/latest
    ports:
      - containerPort: 80
        name: web
kubernetesResources:
  pod:
    securityContext:
      fsGroup: 1001

So when you say “outside the containers” I think we are both referring to the pod level security context? This is what I confirmed Juju was setting with the fsGroup and passing to the statefulset create() API but was not showing up later when inspecting the pods that got stood up. It’s not clear at this stage why that is happening. I may have had a typo in the fsGroup value and it got ignored, but haven’t had a chance to retest.

Ah yeah sorry, I think we’re on the same line, I was thrown because the container level securityContext entries and the mentions of pods. My fault!

Cause this is liable to get lost over Christmas and New Years and blocks deployments of K8S charms onto a few different implementations I’ve filed a bug on launchpad Bug #1909153 “fsGroup unsettable in juju k8s” : Bugs : juju

Thanks for filing the bug, @magicaltrout. I triaged and tagged it as something we will fix if possible this cycle.

@magicaltrout support for pod level security context should already be there as per

version: 3
containers:
  - name: something
    image: something/latest
    ports:
      - containerPort: 80
        name: web
kubernetesResources:
  pod:
    securityContext:
      fsGroup: 1001

Is this not working for you?

I can test again later, I don’t believe I found a combination that works:

isn’t that the same as your spec?

Yeah, looks the same. When I last tested, I confirmed the k8s pod spec template that got set on the statefulset had the correct attributes set. But maybe I should re-test to be sure.

I just did another test. juju version 2.8.7
My charm YAML

version: 3
containers:
  - name: mariadb-k8s
    ports:
    - containerPort: 3361
      protocol: TCP
...
kubernetesResources:
  pod:
    securityContext:
      runAsUser: 1001
      runAsGroup: 1001
      runAsNonRoot: true
      fsGroup: 1001

I deployed the charm and then queried both the statefulset template and workload pod and things were as expected:

$ kubectl get pods/mariadb-k8s-0 -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    juju.io/charm-modified-version: "0"
    juju.io/controller: 4bdfb2d4-422c-4a15-896c-209e3c65b714
    juju.io/model: ac39a35e-9df1-450e-87da-66a0fb564614
    juju.io/unit: mariadb-k8s/0
...
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1001
    runAsGroup: 1001
    runAsNonRoot: true
    runAsUser: 1001
...
$ kubectl get statefulset.apps/mariadb-k8s -o yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    juju-app-uuid: 7d695a9d
    juju.io/charm-modified-version: "0"
    juju.io/controller: 4bdfb2d4-422c-4a15-896c-209e3c65b714
    juju.io/model: ac39a35e-9df1-450e-87da-66a0fb564614
  creationTimestamp: "2021-01-05T01:12:30Z"
  generation: 1
...
spec:
  podManagementPolicy: Parallel
  replicas: 1
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      juju-app: mariadb-k8s
  serviceName: mariadb-k8s-endpoints
  template:
...
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
        runAsGroup: 1001
        runAsNonRoot: true
        runAsUser: 1001
...

Weird. Thanks for checking @wallyworld I’ll test tomorrow and let you know what I see.

Okay, as it works for you I assume I’m doing something stupid:

https://gitlab.com/spiculedata/juju/solr-k8s-charm/-/blob/master/src/charm.py#L73-90

This is my charm test spec.

Pod log:

Error executing 'postInstallation': EACCES: permission denied, mkdir '/bitnami/solr'

There is no set fsGroup at pod or statefulset level :man_shrugging:

I can’t figure it out, this is just a standard 1.19 K8S OVH cluster, happy to give you access to it if it makes life any easier. I guess its probably more likely me doing something foolish in the bootstrap process.