I have a working 3-unit innodb cluster as part of an Openstack model. No problem with that but I have to do some chassis maintenance, so I tried adding a fourth unit in order that I will still have three while working on the old chassis’. The new unit did not join the existing cluster.
When the charm had settled, the error.log on the new unit was not indicating any problem;
2021-10-28T14:55:49.532359Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-10-28T14:55:49.862410Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-10-28T14:55:50.202554Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
2021-10-28T14:55:50.202624Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
2021-10-28T14:55:50.204418Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-10-28T14:55:50.204504Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2021-10-28T14:55:50.239248Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2021-10-28T14:55:50.239384Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.27-0ubuntu0.20.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu).
I then ran the add-unit action, which claimed to succeed;
$ juju run-action mysql-innodb-cluster/leader --wait add-instance address=10.23.130.207
unit-mysql-innodb-cluster-0:
UnitId: mysql-innodb-cluster/0
id: "8864"
results:
outcome: Success
output: None
status: completed
timing:
completed: 2021-10-28 16:03:14 +0000 UTC
enqueued: 2021-10-28 16:02:50 +0000 UTC
started: 2021-10-28 16:02:50 +0000 UTC
But the new unit shows “Cluster is inaccessible from this instance…”. The error.log on the new unit includes;
2021-10-28T15:00:57.748958Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.27-0ubuntu0.20.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu).
2021-10-28T16:03:09.558489Z 10 [ERROR] [MY-011685] [Repl] Plugin group_replication reported: 'The group_replication_group_name option is mandatory'
2021-10-28T16:03:09.558552Z 10 [ERROR] [MY-011660] [Repl] Plugin group_replication reported: 'Unable to start Group Replication on boot'
2021-10-28T16:03:10.065244Z 10 [Warning] [MY-010604] [Repl] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=juju-7008a6-5-lxd-0-relay-bin' to avoid this problem.
2021-10-28T16:03:10.067820Z 10 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''.
2021-10-28T16:03:10.072468Z 10 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.'
2021-10-28T16:03:10.075044Z 10 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Automatically adding IPv4 localhost address to the allowlist. It is mandatory that it is added.'
2021-10-28T16:03:10.075097Z 10 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Automatically adding IPv6 localhost address to the allowlist. It is mandatory that it is added.'
2021-10-28T16:03:10.126970Z 14 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2021-10-28T16:03:12.171834Z 0 [Warning] [MY-013783] [Repl] This member joined a group on which all members do not support replication failover channels integration on Group Replication, as such it did reset its replication failover channels configuration to the default one.
2021-10-28T16:03:12.179248Z 0 [Warning] [MY-013426] [Repl] Plugin group_replication reported: 'Member version is read compatible with the group.'
2021-10-28T16:03:12.179412Z 10 [System] [MY-011511] [Repl] Plugin group_replication reported: 'This server is working as secondary member with primary member address 10.23.130.15:3306.'
2021-10-28T16:03:13.180319Z 0 [ERROR] [MY-013467] [Repl] Plugin group_replication reported: 'No valid or ONLINE members exist to get the missing data from the group. For cloning check if donors of the same version and with clone plugin installed exist. For incremental recovery check if you have donors where the required data was not purged from the binary logs.'
2021-10-28T16:03:13.180486Z 0 [ERROR] [MY-011712] [Repl] Plugin group_replication reported: 'The server was automatically set into read only mode after an error was detected.'
2021-10-28T16:03:13.750313Z 10 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''.
2021-10-28T16:03:14.181812Z 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to 10.23.130.207:3306, 10.23.130.40:3306, 10.23.130.15:3306, 10.23.130.25:3306 on view 16299025927432118:14.'
2021-10-28T16:03:14.182004Z 0 [ERROR] [MY-011486] [Repl] Plugin group_replication reported: 'Message received while the plugin is not ready, message discarded.'
2021-10-28T16:03:14.182055Z 0 [ERROR] [MY-011486] [Repl] Plugin group_replication reported: 'Message received while the plugin is not ready, message discarded.'
I also tried to run the ‘remove-instance’ and ‘rejoin-instance’ actions on the cluster but these also fail.
Please see the error.log and the juju debug-log
Any suggestions on how to proceed please? Many thanks in advance!