Failing Mysql 8 cluster (mysql-innodb-cluster)

Hi guys,

I have had some problems with getting the mysql-innodb-cluster charm working.

In the below example is a simple three node Mysql cluster in LXD containers running on virtual hardware deployed through MAAS.

juju add-model osdev --config default-series=focal --config default-space=os-internal
juju add-machine -n3 --constraints tags=controller
juju deploy -n 3 --to lxd:0,lxd:1,lxd:2 mysql-innodb-cluster --config cluster-name=myCluster

Waiting a few minutes to let everything settle down

App                   Version  Status   Scale  Charm                 Store       Rev  OS      Notes
mysql-innodb-cluster  8.0.22   waiting      3  mysql-innodb-cluster  jujucharms    3  ubuntu  

Unit                     Workload  Agent      Machine  Public address  Ports  Message
mysql-innodb-cluster/0   waiting   idle       0        10.10.4.5              Instance not yet configured for clustering
mysql-innodb-cluster/1*  waiting   executing  1        10.10.4.6              'cluster' incomplete, Not all instances configured for clustering
mysql-innodb-cluster/2   waiting   idle       2        10.10.4.7              Instance not yet configured for clustering

Machine  State    DNS        Inst id      Series  AZ   Message
0        started  10.10.4.5  controller1  focal   az1  Deployed
1        started  10.10.4.6  controller2  focal   az1  Deployed
2        started  10.10.4.7  controller3  focal   az1  Deployed

One of the Units is stuck in executing waiting for the others. The other two units are stuck in waiting for instance cluster configuration.

In the JuJu debug logs i see this:

unit-mysql-innodb-cluster-2: 14:02:59 INFO unit.mysql-innodb-cluster/2.juju-log cluster:0: Invoking reactive handler: reactive/mysql_innodb_cluster_handlers.py:69:send_cluster_connection_info
unit-mysql-innodb-cluster-2: 14:03:00 INFO unit.mysql-innodb-cluster/2.juju-log cluster:0: Invoking reactive handler: hooks/relations/tls-certificates/requires.py:109:broken:certificates
unit-mysql-innodb-cluster-2: 14:03:00 INFO unit.mysql-innodb-cluster/2.juju-log cluster:0: Invoking reactive handler: hooks/relations/mysql-innodb-cluster/peers.py:69:joined:cluster
unit-mysql-innodb-cluster-2: 14:03:00 INFO unit.mysql-innodb-cluster/2.juju-log cluster:0: Invoking reactive handler: hooks/relations/mysql-innodb-cluster/peers.py:75:changed:cluster
unit-mysql-innodb-cluster-2: 14:03:00 INFO unit.mysql-innodb-cluster/2.juju-log cluster:0: Invoking reactive handler: reactive/mysql_innodb_cluster_handlers.py:91:create_remote_cluster_user
unit-mysql-innodb-cluster-2: 14:03:00 WARNING unit.mysql-innodb-cluster/2.juju-log cluster:0: This instance is not yet clustered: cannot determine the cluster status.
unit-mysql-innodb-cluster-2: 14:03:00 WARNING unit.mysql-innodb-cluster/2.juju-log cluster:0: Cannot determine the cluster primary RW node for writes.
unit-mysql-innodb-cluster-1: 14:03:00 WARNING unit.mysql-innodb-cluster/1.cluster-relation-joined All snaps up to date.
unit-mysql-innodb-cluster-1: 14:03:00 INFO unit.mysql-innodb-cluster/1.juju-log cluster:0: Initializing Leadership Layer (is leader)
unit-mysql-innodb-cluster-1: 14:03:00 INFO unit.mysql-innodb-cluster/1.juju-log cluster:0: Coordinator: Initializing coordinator layer
unit-mysql-innodb-cluster-2: 14:03:00 WARNING unit.mysql-innodb-cluster/2.juju-log cluster:0: This instance is not yet clustered: cannot determine the cluster status.
unit-mysql-innodb-cluster-2: 14:03:00 WARNING unit.mysql-innodb-cluster/2.juju-log cluster:0: Cannot determine the cluster primary RW node for writes.
unit-mysql-innodb-cluster-1: 14:03:00 INFO unit.mysql-innodb-cluster/1.juju-log cluster:0: Invoking reactive handler: reactive/mysql_innodb_cluster_handlers.py:69:send_cluster_connection_info
unit-mysql-innodb-cluster-1: 14:03:00 INFO unit.mysql-innodb-cluster/1.juju-log cluster:0: Invoking reactive handler: hooks/relations/mysql-innodb-cluster/peers.py:69:joined:cluster
unit-mysql-innodb-cluster-1: 14:03:00 INFO unit.mysql-innodb-cluster/1.juju-log cluster:0: Invoking reactive handler: hooks/relations/mysql-innodb-cluster/peers.py:75:changed:cluster
unit-mysql-innodb-cluster-1: 14:03:00 INFO unit.mysql-innodb-cluster/1.juju-log cluster:0: Invoking reactive handler: hooks/relations/tls-certificates/requires.py:109:broken:certificates
unit-mysql-innodb-cluster-1: 14:03:00 INFO unit.mysql-innodb-cluster/1.juju-log cluster:0: Invoking reactive handler: reactive/mysql_innodb_cluster_handlers.py:91:create_remote_cluster_user
unit-mysql-innodb-cluster-2: 14:03:00 INFO unit.mysql-innodb-cluster/2.juju-log cluster:0: Invoking reactive handler: reactive/mysql_innodb_cluster_handlers.py:198:signal_clustered

I cannot see anything obvious about what is or isn’t happening here. Cannot find anything that i am missing in the documentation.

It worked a few days ago when deploying my OpenStack environment and then suddenly stopped.

Any ideas?

FWIW, I performed a quick test of this and it worked. The underlying machines are KVM-based MAAS nodes. Maybe it is a Juju spaces issue; I didn’t use them.

juju add-model --config default-series=focal mysql8
juju set-model-constraints tags=virtual
juju deploy -n 3 --to lxd,lxd,lxd mysql-innodb-cluster --config cluster-name=myCluster

Output to juju status:

Model   Controller  Cloud/Region    Version  SLA          Timestamp
mysql8  xxxxxx      xxxxxx/default  2.8.6    unsupported  17:16:16Z

App                   Version  Status  Scale  Charm                 Store       Rev  OS      Notes
mysql-innodb-cluster  8.0.22   active      3  mysql-innodb-cluster  jujucharms    3  ubuntu  

Unit                     Workload  Agent  Machine  Public address  Ports  Message
mysql-innodb-cluster/0*  active    idle   0/lxd/0  xxx.xxx.114.52          Unit is ready: Mode: R/W
mysql-innodb-cluster/1   active    idle   1/lxd/0  xxx.xxx.114.53          Unit is ready: Mode: R/O
mysql-innodb-cluster/2   active    idle   2/lxd/0  xxx.xxx.114.54          Unit is ready: Mode: R/O

Machine  State    DNS            Inst id              Series  AZ       Message
0        started  xxx.xxx.114.38  virt-node-02         focal   default  Deployed
0/lxd/0  started  xxx.xxx.114.52  juju-a67ad0-0-lxd-0  focal   default  Container started
1        started  xxx.xxx.114.50  virt-node-03         focal   default  Deployed
1/lxd/0  started  xxx.xxx.114.53  juju-a67ad0-1-lxd-0  focal   default  Container started
2        started  xxx.xxx.114.51  virt-node-04         focal   default  Deployed
2/lxd/0  started  xxx.xxx.114.54  juju-a67ad0-2-lxd-0  focal   default  Container started