Hi guys,
I have had some problems with getting the mysql-innodb-cluster charm working.
In the below example is a simple three node Mysql cluster in LXD containers running on virtual hardware deployed through MAAS.
juju add-model osdev --config default-series=focal --config default-space=os-internal
juju add-machine -n3 --constraints tags=controller
juju deploy -n 3 --to lxd:0,lxd:1,lxd:2 mysql-innodb-cluster --config cluster-name=myCluster
Waiting a few minutes to let everything settle down
App Version Status Scale Charm Store Rev OS Notes
mysql-innodb-cluster 8.0.22 waiting 3 mysql-innodb-cluster jujucharms 3 ubuntu
Unit Workload Agent Machine Public address Ports Message
mysql-innodb-cluster/0 waiting idle 0 10.10.4.5 Instance not yet configured for clustering
mysql-innodb-cluster/1* waiting executing 1 10.10.4.6 'cluster' incomplete, Not all instances configured for clustering
mysql-innodb-cluster/2 waiting idle 2 10.10.4.7 Instance not yet configured for clustering
Machine State DNS Inst id Series AZ Message
0 started 10.10.4.5 controller1 focal az1 Deployed
1 started 10.10.4.6 controller2 focal az1 Deployed
2 started 10.10.4.7 controller3 focal az1 Deployed
One of the Units is stuck in executing waiting for the others. The other two units are stuck in waiting for instance cluster configuration.
In the JuJu debug logs i see this:
unit-mysql-innodb-cluster-2: 14:02:59 INFO unit.mysql-innodb-cluster/2.juju-log cluster:0: Invoking reactive handler: reactive/mysql_innodb_cluster_handlers.py:69:send_cluster_connection_info
unit-mysql-innodb-cluster-2: 14:03:00 INFO unit.mysql-innodb-cluster/2.juju-log cluster:0: Invoking reactive handler: hooks/relations/tls-certificates/requires.py:109:broken:certificates
unit-mysql-innodb-cluster-2: 14:03:00 INFO unit.mysql-innodb-cluster/2.juju-log cluster:0: Invoking reactive handler: hooks/relations/mysql-innodb-cluster/peers.py:69:joined:cluster
unit-mysql-innodb-cluster-2: 14:03:00 INFO unit.mysql-innodb-cluster/2.juju-log cluster:0: Invoking reactive handler: hooks/relations/mysql-innodb-cluster/peers.py:75:changed:cluster
unit-mysql-innodb-cluster-2: 14:03:00 INFO unit.mysql-innodb-cluster/2.juju-log cluster:0: Invoking reactive handler: reactive/mysql_innodb_cluster_handlers.py:91:create_remote_cluster_user
unit-mysql-innodb-cluster-2: 14:03:00 WARNING unit.mysql-innodb-cluster/2.juju-log cluster:0: This instance is not yet clustered: cannot determine the cluster status.
unit-mysql-innodb-cluster-2: 14:03:00 WARNING unit.mysql-innodb-cluster/2.juju-log cluster:0: Cannot determine the cluster primary RW node for writes.
unit-mysql-innodb-cluster-1: 14:03:00 WARNING unit.mysql-innodb-cluster/1.cluster-relation-joined All snaps up to date.
unit-mysql-innodb-cluster-1: 14:03:00 INFO unit.mysql-innodb-cluster/1.juju-log cluster:0: Initializing Leadership Layer (is leader)
unit-mysql-innodb-cluster-1: 14:03:00 INFO unit.mysql-innodb-cluster/1.juju-log cluster:0: Coordinator: Initializing coordinator layer
unit-mysql-innodb-cluster-2: 14:03:00 WARNING unit.mysql-innodb-cluster/2.juju-log cluster:0: This instance is not yet clustered: cannot determine the cluster status.
unit-mysql-innodb-cluster-2: 14:03:00 WARNING unit.mysql-innodb-cluster/2.juju-log cluster:0: Cannot determine the cluster primary RW node for writes.
unit-mysql-innodb-cluster-1: 14:03:00 INFO unit.mysql-innodb-cluster/1.juju-log cluster:0: Invoking reactive handler: reactive/mysql_innodb_cluster_handlers.py:69:send_cluster_connection_info
unit-mysql-innodb-cluster-1: 14:03:00 INFO unit.mysql-innodb-cluster/1.juju-log cluster:0: Invoking reactive handler: hooks/relations/mysql-innodb-cluster/peers.py:69:joined:cluster
unit-mysql-innodb-cluster-1: 14:03:00 INFO unit.mysql-innodb-cluster/1.juju-log cluster:0: Invoking reactive handler: hooks/relations/mysql-innodb-cluster/peers.py:75:changed:cluster
unit-mysql-innodb-cluster-1: 14:03:00 INFO unit.mysql-innodb-cluster/1.juju-log cluster:0: Invoking reactive handler: hooks/relations/tls-certificates/requires.py:109:broken:certificates
unit-mysql-innodb-cluster-1: 14:03:00 INFO unit.mysql-innodb-cluster/1.juju-log cluster:0: Invoking reactive handler: reactive/mysql_innodb_cluster_handlers.py:91:create_remote_cluster_user
unit-mysql-innodb-cluster-2: 14:03:00 INFO unit.mysql-innodb-cluster/2.juju-log cluster:0: Invoking reactive handler: reactive/mysql_innodb_cluster_handlers.py:198:signal_clustered
I cannot see anything obvious about what is or isn’t happening here. Cannot find anything that i am missing in the documentation.
It worked a few days ago when deploying my OpenStack environment and then suddenly stopped.
Any ideas?