Me again,
If I deploy Grafana or Prometheus into my Cinder backed OVH Kubernetes cluster, the mounted filesystems get a permission denied error.
Pretty common for both it seems judging by the google and the course of action seems to be:
Setting
securityContext:
runAsUser: 472
fsGroup: 472
I don’t see either in the securityGroup in the Kubernetes spec 3 tests or elsewhere. Can I set that, or is there an alternative course of action?
I think this might work (I’ve not tested it):
version: 3
containers:
- name: something
image: something/latest
ports:
- containerPort: 80
name: fred
kubernetes:
securityContext:
runAsUser: 472
fsGroup: 472
Yeah I tried this in v2 and v3 spec and I get an error:
config_content = self._build_grafana_ini()
spec = {
"version":3,
"containers": [
{
"name": self.app.name,
"imageDetails": {"imagePath": "grafana/grafana:7.2.1-ubuntu"},
"ports": [{"containerPort": port, "protocol": "TCP"}],
"volumeConfig": {
"name": "grafana-config-ini",
"mountPath": "/etc/grafana",
"files": {"grafana.ini": config_content},
},
"kubernetes": {
"securityContext":{
#"runAsNonRoot": true
"runAsUser": 472,
"fsGroup": 472,
},
"readinessProbe": {
"httpGet": {"path": "/api/health", "port": port},
"initialDelaySeconds": 10,
"timeoutSeconds": 30,
},
},
}
]
}
ops.model.ModelError: b'ERROR json: unknown field "fsGroup"\n'
Which suggests to me that the securityContext tag is in the correct place but it hasn’t got a clue what to do with the fsGroup value.
Ah sorry, there’s a difference between container security context and pod security context.
This should work.
version: 3
containers:
- name: something
image: something/latest
ports:
- containerPort: 80
name: web
kubernetesResources:
pod:
securityContext:
runAsUser: 32
fsGroup: 32
Sorry @wallyworld still not quite sussed this.
Here’s my spec:
spec = {
"version": 3,
"kubernetesResources":{
"pod":{
"securityContext":{
"fsGroup": 1001,
"runAsUser": 1001,
"runAsGroup":1001,
}
}
},
"containers": [
{
"name": self.app.name,
"imageDetails": image_details,
"imagePullPolicy": "Always",
"ports": ports,
},
],
}
it validated and deployed but still fails. And if I edit the podspec on the cluster it says:
containers:
- image: bitnami/solr:8.7.0-debian-10-r31
imagePullPolicy: Always
name: solr
ports:
- containerPort: 8983
name: solr
protocol: TCP
resources: {}
securityContext:
allowPrivilegeEscalation: true
readOnlyRootFilesystem: false
runAsNonRoot: false
Which clearly doesn’t have the stuff I specified in it.
I just deployed a small charm with the fsGroup set and via some debugging confirmed that the values are set in the statefulset Spec.Template when the statefulset create API is called.
We’ll have to do some investigation as to why aspects of the Spec.Template are not being applied as requested.
{
...
"containers": [
{
"name": "mariadb-k8s",
"image": "mariadb",
"ports": [
{
"containerPort": 3306,
"protocol": "TCP"
}
],
"env": [
{
"name": "MYSQL_DATABASE",
"value": "database"
},
{
"name": "MYSQL_PASSWORD",
"value": "password"
},
{
"name": "MYSQL_ROOT_PASSWORD",
"value": "root"
},
{
"name": "MYSQL_USER",
"value": "admin"
},
{
"name": "NODE_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "spec.nodeName"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "juju-data-dir",
"mountPath": "/var/lib/juju"
},
{
"name": "juju-data-dir",
"mountPath": "/usr/bin/juju-run",
"subPath": "tools/jujud"
},
{
"name": "mariadb-k8s-configurations-config",
"mountPath": "/etc/mysql/conf.d"
}
],
"securityContext": {
"runAsNonRoot": false,
"readOnlyRootFilesystem": false,
"allowPrivilegeEscalation": true
}
}
],
"serviceAccountName": "mariadb-k8s",
"automountServiceAccountToken": true,
"securityContext": {
"fsGroup": 2
}
}
Hmm fair enough & thanks for checking. The other thing I see people do is setup an init container and chown the mount directory space prior to the main container coming up. I tried that also earlier but it appears the volumes specified in metadata.yaml aren’t mounted in init pods, is there anyway to get that pvc mounted in the init pod?
Currently any Juju storage pv is only mounted in the workload container.
We could look at if it makes sense to also mount them in an init container to allow the sort of initialisation you are talking about. But then there’s issues reclaim policy to consider etc. I’d also like to understand why the fsGroup is not being applied.
Yeah,
I don’t think either of these methods are particularly weird for non-root running containers, but the setting of fsGroup and the runAsUser etc seem to be the correct way of going about it. chowning a bunch of stuff in an init container is particularly unelegant.
This also obviously affects any non-root charm in the charmstore, I saw it with, I believe grafana as well the other day when testing, so it’ll be cool to get this figured out and stuck into various K8S charms that currently mount volumes with the wrong permissions.
So did a bit more digging @wallyworld
I deployed my charm on microk8s and it worked without modification, which is what I was expecting as ya’ll deploy charms with pvc’s without bother.
So I then edited the statefulset and realised there are 2 securityContexts, and this setting only seems to stick in the one outside of the containers: block at spec level.
As soon as I added:
securityContext:
fsGroup: 1001
To the outer context, the pod came to life and fixed the permissions error I was facing.
So, can you set higher level kubernetes resources?
Thanks
Tom
You are referring to the PodSecurityContext value on the statefulset spec template right?
type StatefulSetSpec struct {
Template v1.PodTemplateSpec
}
type PodTemplateSpec struct {
metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
Spec PodSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
}
type PodSpec struct {
Volumes []Volume `json:"volumes,omitempty" patchStrategy:"merge,retainKeys" patchMergeKey:"name" protobuf:"bytes,1,rep,name=volumes"`
InitContainers []Container `json:"initContainers,omitempty" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,20,rep,name=initContainers"`
Containers []Container `json:"containers" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,2,rep,name=containers"`
// SecurityContext holds pod-level security attributes and common container settings.
// Optional: Defaults to empty. See type description for default values of each field.
// +optional
SecurityContext *PodSecurityContext `json:"securityContext,omitempty" protobuf:"bytes,14,opt,name=securityContext"`
}
The SecurityContext in the PodSpec above is what the securityContext passed to Juju is set from
version: 3
containers:
- name: something
image: something/latest
ports:
- containerPort: 80
name: web
kubernetesResources:
pod:
securityContext:
fsGroup: 1001
So when you say “outside the containers” I think we are both referring to the pod level security context? This is what I confirmed Juju was setting with the fsGroup and passing to the statefulset create() API but was not showing up later when inspecting the pods that got stood up. It’s not clear at this stage why that is happening. I may have had a typo in the fsGroup value and it got ignored, but haven’t had a chance to retest.
Ah yeah sorry, I think we’re on the same line, I was thrown because the container level securityContext entries and the mentions of pods. My fault!
Cause this is liable to get lost over Christmas and New Years and blocks deployments of K8S charms onto a few different implementations I’ve filed a bug on launchpad Bug #1909153 “fsGroup unsettable in juju k8s” : Bugs : juju
Thanks for filing the bug, @magicaltrout. I triaged and tagged it as something we will fix if possible this cycle.
@magicaltrout support for pod level security context should already be there as per
version: 3
containers:
- name: something
image: something/latest
ports:
- containerPort: 80
name: web
kubernetesResources:
pod:
securityContext:
fsGroup: 1001
Is this not working for you?
I can test again later, I don’t believe I found a combination that works:
isn’t that the same as your spec?
Yeah, looks the same. When I last tested, I confirmed the k8s pod spec template that got set on the statefulset had the correct attributes set. But maybe I should re-test to be sure.
I just did another test. juju version 2.8.7
My charm YAML
version: 3
containers:
- name: mariadb-k8s
ports:
- containerPort: 3361
protocol: TCP
...
kubernetesResources:
pod:
securityContext:
runAsUser: 1001
runAsGroup: 1001
runAsNonRoot: true
fsGroup: 1001
I deployed the charm and then queried both the statefulset template and workload pod and things were as expected:
$ kubectl get pods/mariadb-k8s-0 -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
juju.io/charm-modified-version: "0"
juju.io/controller: 4bdfb2d4-422c-4a15-896c-209e3c65b714
juju.io/model: ac39a35e-9df1-450e-87da-66a0fb564614
juju.io/unit: mariadb-k8s/0
...
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1001
runAsGroup: 1001
runAsNonRoot: true
runAsUser: 1001
...
$ kubectl get statefulset.apps/mariadb-k8s -o yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
annotations:
juju-app-uuid: 7d695a9d
juju.io/charm-modified-version: "0"
juju.io/controller: 4bdfb2d4-422c-4a15-896c-209e3c65b714
juju.io/model: ac39a35e-9df1-450e-87da-66a0fb564614
creationTimestamp: "2021-01-05T01:12:30Z"
generation: 1
...
spec:
podManagementPolicy: Parallel
replicas: 1
revisionHistoryLimit: 0
selector:
matchLabels:
juju-app: mariadb-k8s
serviceName: mariadb-k8s-endpoints
template:
...
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1001
runAsGroup: 1001
runAsNonRoot: true
runAsUser: 1001
...
Okay, as it works for you I assume I’m doing something stupid:
https://gitlab.com/spiculedata/juju/solr-k8s-charm/-/blob/master/src/charm.py#L73-90
This is my charm test spec.
Pod log:
Error executing 'postInstallation': EACCES: permission denied, mkdir '/bitnami/solr'
There is no set fsGroup at pod or statefulset level 
I can’t figure it out, this is just a standard 1.19 K8S OVH cluster, happy to give you access to it if it makes life any easier. I guess its probably more likely me doing something foolish in the bootstrap process.