Trying to deploy openstack using sunbeam (2023.02) on a multi node machine, this machine has a 2TB ssd for the OS (ubuntu 22.04) and 2 5TB HDD for ceph. It has 128GB of memory adn 48 cores as well as 2 network interfaces.
When trying to deploy, it often fails on mysql installation, here is an example of the logs. I was wondering what can I do to recover from this. I tried to re-install (10+ times) but it always fails at one of the mysql instances.
024-04-01T21:59:00.830Z [container-agent]
2024-04-01T21:59:06.819Z [container-agent] 2024-04-01 21:59:06 ERROR juju-log database-peers:13: Failed to connect to MySQL with mysqlsh
2024-04-01T21:59:06.819Z [container-agent] Traceback (most recent call last):
2024-04-01T21:59:06.819Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/src/mysql_k8s_helpers.py", line 666, in _run_mysqlsh_script
2024-04-01T21:59:06.819Z [container-agent] stdout, _ = process.wait_output()
2024-04-01T21:59:06.819Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/ops/pebble.py", line 1540, in wait_output
2024-04-01T21:59:06.819Z [container-agent] raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
2024-04-01T21:59:06.819Z [container-agent] ops.pebble.ExecError: non-zero exit code 1 executing ['/usr/bin/mysqlsh', '--no-wizard', '--python', '--verbose=1', '-f', '/tmp/script.py', ';', 'rm', '/tmp/script.py'], stdout='', stderr='Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nverbose: 2024-04-01T21:59:06Z: Loading startup files...\nverbose: 2024-04-01T21:59:06Z: Loading plugins...\nverbose: 2024-04-01T21:59:06Z: Connecting to MySQL at: serverconfig@cinder-mysql-0.cinder-mysql-endpoints.openstack.svc.cluster.local\nTraceback (most recent call last):\n File "<string>", line 1, in <module>\nmysqlsh.DBError: MySQL Error (1045): Shell.connect: Access denied for user \'serverconfig\'@\'cinder-mysql-0.cinder-mysql-endpoints.openstack.svc.cluster.loca\' (using password: YES)\n'
2024-04-01T21:59:06.819Z [container-agent]
2024-04-01T21:59:06.819Z [container-agent] During handling of the above exception, another exception occurred:
2024-04-01T21:59:06.819Z [container-agent]
2024-04-01T21:59:06.819Z [container-agent] Traceback (most recent call last):
2024-04-01T21:59:06.819Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 2501, in check_mysqlsh_connection
2024-04-01T21:59:06.819Z [container-agent] self._run_mysqlsh_script("\n".join(connect_commands))
2024-04-01T21:59:06.819Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/src/mysql_k8s_helpers.py", line 669, in _run_mysqlsh_script
2024-04-01T21:59:06.819Z [container-agent] raise MySQLClientError(e.stderr)
2024-04-01T21:59:06.819Z [container-agent] charms.mysql.v0.mysql.MySQLClientError: Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory
2024-04-01T21:59:06.819Z [container-agent] verbose: 2024-04-01T21:59:06Z: Loading startup files...
2024-04-01T21:59:06.819Z [container-agent] verbose: 2024-04-01T21:59:06Z: Loading plugins...
2024-04-01T21:59:06.819Z [container-agent] verbose: 2024-04-01T21:59:06Z: Connecting to MySQL at: serverconfig@cinder-mysql-0.cinder-mysql-endpoints.openstack.svc.cluster.local
2024-04-01T21:59:06.819Z [container-agent] Traceback (most recent call last):
2024-04-01T21:59:06.819Z [container-agent] File "<string>", line 1, in <module>
2024-04-01T21:59:06.819Z [container-agent] mysqlsh.DBError: MySQL Error (1045): Shell.connect: Access denied for user 'serverconfig'@'cinder-mysql-0.cinder-mysql-endpoints.openstack.svc.cluster.loca' (using password: YES)
2024-04-01T21:59:06.819Z [container-agent]
2024-04-01T21:59:06.876Z [container-agent] 2024-04-01 21:59:06 ERROR juju-log database-peers:13: Uncaught exception while in charm code:
2024-04-01T21:59:06.876Z [container-agent] Traceback (most recent call last):
2024-04-01T21:59:06.876Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/./src/charm.py", line 770, in <module>
2024-04-01T21:59:06.876Z [container-agent] main(MySQLOperatorCharm)
2024-04-01T21:59:06.876Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/ops/main.py", line 454, in main
2024-04-01T21:59:06.876Z [container-agent] framework.reemit()
2024-04-01T21:59:06.876Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/ops/framework.py", line 863, in reemit
2024-04-01T21:59:06.876Z [container-agent] self._reemit()
2024-04-01T21:59:06.876Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/ops/framework.py", line 943, in _reemit
2024-04-01T21:59:06.876Z [container-agent] custom_handler(event)
2024-04-01T21:59:06.876Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/./src/charm.py", line 570, in _on_mysql_pebble_ready
2024-04-01T21:59:06.876Z [container-agent] self._reconcile_pebble_layer(container)
2024-04-01T21:59:06.876Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/./src/charm.py", line 339, in _reconcile_pebble_layer
2024-04-01T21:59:06.876Z [container-agent] self._mysql.wait_until_mysql_connection()
2024-04-01T21:59:06.876Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/tenacity/__init__.py", line 289, in wrapped_f
2024-04-01T21:59:06.876Z [container-agent] return self(f, *args, **kw)
2024-04-01T21:59:06.876Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/tenacity/__init__.py", line 379, in __call__
2024-04-01T21:59:06.876Z [container-agent] do = self.iter(retry_state=retry_state)
2024-04-01T21:59:06.876Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/tenacity/__init__.py", line 325, in iter
2024-04-01T21:59:06.876Z [container-agent] raise retry_exc.reraise()
2024-04-01T21:59:06.876Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/tenacity/__init__.py", line 158, in reraise
2024-04-01T21:59:06.876Z [container-agent] raise self.last_attempt.result()
2024-04-01T21:59:06.876Z [container-agent] File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2024-04-01T21:59:06.876Z [container-agent] return self.__get_result()
2024-04-01T21:59:06.876Z [container-agent] File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2024-04-01T21:59:06.876Z [container-agent] raise self._exception
2024-04-01T21:59:06.876Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/tenacity/__init__.py", line 382, in __call__
2024-04-01T21:59:06.876Z [container-agent] result = fn(*args, **kwargs)
2024-04-01T21:59:06.876Z [container-agent] File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/src/mysql_k8s_helpers.py", line 233, in wait_until_mysql_connection
2024-04-01T21:59:06.876Z [container-agent] raise MySQLServiceNotRunningError("Connection with mysqlsh not possible")
2024-04-01T21:59:06.876Z [container-agent] charms.mysql.v0.mysql.MySQLServiceNotRunningError: Connection with mysqlsh not possible
2024-04-01T21:59:07.160Z [container-agent] 2024-04-01 21:59:07 ERROR juju.worker.uniter.operation runhook.go:180 hook "database-peers-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
2024-04-01T21:59:07.162Z [container-agent] 2024-04-01 21:59:07 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-changed" hook