Hi Team,
I am getting the below error message while fetching the image details from glance. I am facing this issue randomly.
Need your help to fix this issue.
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines [req-3587c4a4-aec3-4ae3-bf51-35bfa9174626 93a2a385895e4a1890b25dae0187d391 030cb4d8d07c4824a77862b27482fd38 - 1a7d8253d1b2492cb31316a2b5a52ce4 1a7d8253d1b2492cb31316a2b5a52ce4] Database connection was found disconnected; reconnecting: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
(Background on this error at: https://sqlalche.me/e/14/e3q8)
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines self.dialect.do_execute(
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 148, in execute
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines result = self._query(query)
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 310, in _query
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines conn.query(q)
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 548, in query
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 775, in _read_query_result
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines result.read()
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1156, in read
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet()
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 692, in _read_packet
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines packet_header = self._read_bytes(4)
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 748, in _read_bytes
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines raise err.OperationalError(
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines The above exception was the direct cause of the following exception:
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines connection.scalar(select(1))
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1171, in scalar
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines return self.execute(object_, *multiparams, **params).scalar()
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1289, in execute
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines ret = self._execute_context(
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines self._handle_dbapi_exception(
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2024, in _handle_dbapi_exception
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines util.raise_(newraise, with_traceback=exc_info[2], from_=e)
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines raise exception
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines self.dialect.do_execute(
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 148, in execute
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines result = self._query(query)
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 310, in _query
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines conn.query(q)
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 548, in query
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 775, in _read_query_result
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines result.read()
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1156, in read
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet()
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 692, in _read_packet
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines packet_header = self._read_bytes(4)
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 748, in _read_bytes
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines raise err.OperationalError(
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines [SQL: SELECT 1]
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines (Background on this error at: https://sqlalche.me/e/14/e3q8)
2023-05-15 07:16:04.618 105831 ERROR oslo_db.sqlalchemy.engines
root@juju-717d4e-4-lxd-1:/var/log/glance#
juju status
root@opc1-maas-openstack:~# juju status mysql-innodb-cluster
Model Controller Cloud/Region Version SLA Timestamp
openstack maas-controller maas-one/default 2.9.38 unsupported 14:42:23+05:30
App Version Status Scale Charm Channel Rev Exposed Message
mysql-innodb-cluster 8.0.33 active 3 mysql-innodb-cluster 8.0/stable 39 no Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
Unit Workload Agent Machine Public address Ports Message
mysql-innodb-cluster/0 active idle 3/lxd/4 10.85.5.94 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/1* active idle 4/lxd/4 10.85.5.78 Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/2 active idle 5/lxd/4 10.85.5.104 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
Machine State Address Inst id Series AZ Message
3 started 10.85.2.41 controller-1-nx-opc1-b4-27-11 focal default Deployed
3/lxd/4 started 10.85.5.94 juju-717d4e-3-lxd-4 focal default Container started
4 started 10.85.2.69 controller-2-nx-opc1-b3-39-12 focal default Deployed
4/lxd/4 started 10.85.5.78 juju-717d4e-4-lxd-4 focal default Container started
5 started 10.85.2.62 controller-3-nx-opc1-b2-24-13 focal default Deployed
5/lxd/4 started 10.85.5.104 juju-717d4e-5-lxd-4 focal default Container started
Got the below warning from mysql-innodb-cluster units.
unit-2:
root@juju-717d4e-5-lxd-4:~# grep -ir "Shutting down an outgoing connection" /var/log/mysql/error.log
2023-05-15T07:00:26.544308Z 0 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Shutting down an outgoing connection. This happens because something might be wrong on a bi-directional connection to node 10.85.5.78:33061. Please check the connection status to this member'
unit-1:
root@juju-717d4e-4-lxd-4:~# grep -ir "Shutting down an outgoing connection" /var/log/mysql/error.log
2023-05-15T06:58:18.230612Z 0 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Shutting down an outgoing connection. This happens because something might be wrong on a bi-directional connection to node 10.85.5.94:33061. Please check the connection status to this member'
unit-0:
root@juju-717d4e-3-lxd-4:~# grep -ir "Shutting down an outgoing connection" /var/log/mysql/error.log
2023-05-15T06:59:44.517749Z 0 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Shutting down an outgoing connection. This happens because something might be wrong on a bi-directional connection to node 10.85.5.78:33061. Please check the connection status to this member'