Juju OpenStack MySQL Cluster innaccessible after Power Outage

charm OpenStack
1

Hi all, I’ve been following the OpenStack Charms Deployment guide. Yesterday I managed to have an OpenStack cloud completely working, but our data center had a Power Outage and the MySQL InnoDB Cluster is stuck in the following state:

mysql-innodb-cluster/0*        blocked   idle   0/lxd/0  10.12.31.222                    Cluster is inaccessible from this instance. Pleas
e check logs for details.
mysql-innodb-cluster/1         blocked   idle   1/lxd/0  10.12.31.224                    Cluster is inaccessible from this instance. Pleas
e check logs for details.
mysql-innodb-cluster/2         blocked   idle   2/lxd/0  10.12.31.223                    Cluster is inaccessible from this instance. Pleas
e check logs for details.

I’ve read the Managing Power Events page and tried all these commands:

$ juju run mysql-innodb-cluster/0 reboot-cluster-from-complete-outage
Running operation 179 with 1 task
  - task 180 on unit-mysql-innodb-cluster-0

Waiting for task 180...
ERROR timed out waiting for results from: unit mysql-innodb-cluster/0

$ juju run mysql-innodb-cluster/1 reboot-cluster-from-complete-outage
Running operation 185 with 1 task
  - task 186 on unit-mysql-innodb-cluster-1

Waiting for task 186...
Action id 186 failed: Reboot cluster from complete outage failed.
output: |+
  Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory
  Restoring the default cluster from complete outage...

  Traceback (most recent call last):
    File "<string>", line 2, in <module>
  RuntimeError: Dba.reboot_cluster_from_complete_outage: The active session instance (10.12.31.224:3306) isn't the most updated in comparison with the ONLINE instances of the Cluster's metadata. Please use the most up to date instance: '10.12.31.222:3306'.

traceback: |
  Traceback (most recent call last):
    File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/charm/actions/reboot-cluster-from-complete-outage", line 164, in reboot_cluster_from_complete_outage
      output = instance.reboot_cluster_from_complete_outage()
    File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 947, in reboot_cluster_from_complete_outage
      raise e
    File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 935, in reboot_cluster_from_complete_outage
      output = self.run_mysqlsh_script(_script).decode("UTF-8")
    File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 1723, in run_mysqlsh_script
      return subprocess.check_output(cmd, stderr=subprocess.PIPE)
    File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/usr/lib/python3.10/subprocess.py", line 526, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['/snap/bin/mysqlsh', '--no-wizard', '--python', '-f', '/root/snap/mysql-shell/common/tmp8_l6tvzc.py']' returned non-zero exit status 1.

$ juju exec --unit mysql-innodb-cluster/leader leader-get mysql.passwd
ERROR timed out waiting for results from: unit mysql-innodb-cluster/0

Since I can’t get password to connect to the cluster I couldn’t try solutions like this.

Thanks in advance for any help you can give me!

2

Why you cant get a password?

3

Hi @mario-chirinos, as you can see in the commands I pasted, the juju exec --unit mysql-innodb-cluster/leader leader-get mysql.passwd hangs and times out.

4

can you attach the complete output of juju status using paste bin
https://pastebin.ubuntu.com/

5

Sure, you can see that here.

6

@mario-chirinos could you take a look into the previous pastebin?

7

was your cloud working before?

8

Yes, it was

9

are you still not able to get the mysql password?

10

No @mario-chirinos, I couldn’t get it. I’m in the same position I described earlier.