Postgres K8S healthcheck failure

juju
#1

Alrighty,

Sorry @stub any idea on the healthcheck stuff?

I’m deploying the postgres charm and I’ve done plain and with a loadbalancer out of curiosity and it never comes alive:

   Normal   Pulled                  <invalid>                      kubelet                  Successfully pulled image "stubcan/pgcharm:edge" in 1.844038253s
  Normal   Created                 <invalid>                      kubelet                  Created container postgresql
  Normal   Started                 <invalid>                      kubelet                  Started container postgresql
  Warning  Unhealthy               <invalid> (x9 over <invalid>)  kubelet                  Readiness probe failed: dial tcp 10.2.1.26:5432: connect: connection refused

Success. You can now start the database server using:

    pg_ctlcluster 12 main start

Ver Cluster Port Status Owner    Data directory      Log file
12  main    5432 down   postgres /srv/pgdata/12/main /var/log/postgresql/postgresql-12-main.log
2020-12-09 01:00:41,433     INFO: Updating PostgreSQL configuration in /srv/pgconf/12/main/conf.d/juju_charm.conf

Internally it gets that far, but no further.

#2

My guess is you hitting Bug #1903625 “Redeploys to same model fail” : Bugs : charm-k8s-postgresql , with Bug #1903623 “unit names do not always map to pod names” : Bugs : juju being the cause. I think I have a work around (and an improvement) to address this, but it hasn’t been implemented yet.

Work around is to only deploy the charm into a fresh model.

If this isn’t the problem, I’m going to need to land changes with more logging. Assuming this is a single unit, after the pod determining that it should be the master the next step is to log that it is starting PostgreSQL.

#3

Ah okay,

Looks like you were correct.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
  name: default-operator
  namespace: training
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - patch
- apiGroups:
  - ""
  resources:
  - pods/exec
  verbs:
  - create
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: role-test-account-binding
 namespace: training
subjects:
- kind: ServiceAccount
  name: default
  namespace: training
roleRef:
 kind: Role
 name: default-operator
 apiGroup: rbac.authorization.k8s.io

I had to create this role and role binding and apply it whilst the charm was deploying because over here Kubernetes API warnings I can’t tell whether its the charm developers issue or juju’s issue but its using the default service account and it hasn’t got enough permissions to do what your charm asks. That said right at the end it does a patch apparantly and the postgres operator doesn’t have permissions for that either, so I guess it doesn’t make much difference.

Up and running since I chucked that serviceaccount role in though.

Thanks!

1 Like