microstack broken after "sudo reboot now"

juju
1

I follow https://microstack.run/docs/single-node to setup a microstack on PC, everything works well. But when I run “sudo reboot now”, after reboot, the output of “juju status -m openstack” shows:

Unit                         Workload  Agent  Address     Ports  Message
certificate-authority/0*     active    idle   10.1.5.154         
cinder-ceph-mysql-router/0*  active    idle   10.1.5.130         
cinder-ceph/0*               blocked   idle   10.1.5.143         (ceph) integration missing
cinder-mysql-router/0*       active    idle   10.1.5.132         
cinder-mysql/0*              active    idle   10.1.5.185         Primary
cinder/0*                    waiting   idle   10.1.5.148         (workload) Not all relations are ready
glance-mysql-router/0*       active    idle   10.1.5.156         
glance-mysql/0*              active    idle   10.1.5.159         Primary
glance/0*                    active    idle   10.1.5.157         
horizon-mysql-router/0*      active    idle   10.1.5.176         
horizon-mysql/0*             active    idle   10.1.5.146         Primary
horizon/0*                   active    idle   10.1.5.129         
keystone-mysql-router/0*     error     idle   10.1.5.187         hook failed: "start"
keystone-mysql/0*            blocked   idle   10.1.5.179         failed to recover cluster.
keystone/0*                  waiting   idle   10.1.5.181         (workload) Not all relations are ready
neutron-mysql-router/0*      error     idle   10.1.5.191         hook failed: "start"
neutron-mysql/0*             blocked   idle   10.1.5.180         failed to recover cluster.
neutron/0*                   blocked   idle   10.1.5.135         (container:neutron-server) healthcheck failed: online
nova-api-mysql-router/0*     active    idle   10.1.5.173         
nova-cell-mysql-router/0*    active    idle   10.1.5.134         
nova-mysql-router/0*         active    idle   10.1.5.167         
nova-mysql/0*                active    idle   10.1.5.153         Primary
nova/0*                      waiting   idle   10.1.5.150         (container:nova-scheduler) service not ready
ovn-central/0*               active    idle   10.1.5.152         
ovn-relay/0*                 active    idle   10.1.5.177         
placement-mysql-router/0*    active    idle   10.1.5.131         
placement-mysql/0*           active    idle   10.1.5.168         Primary
placement/0*                 active    idle   10.1.5.184         
rabbitmq/0*                  active    idle   10.1.5.138         
traefik/0*                   active    idle   10.1.5.142

keystone-mysql-router/0* and neutron-mysql-router/0* are error with hook failed: “start”. How to solve this problem? Thanks a lot.

2

After I retry “sudo reboot now”, more unit errors:

Unit                         Workload  Agent  Address     Ports  Message
certificate-authority/0*     active    idle   10.1.5.188         
cinder-ceph-mysql-router/0*  error     idle   10.1.5.174         hook failed: "start"
cinder-ceph/0*               blocked   idle   10.1.5.156         (ceph) integration missing
cinder-mysql-router/0*       error     idle   10.1.5.185         hook failed: "start"
cinder-mysql/0*              blocked   idle   10.1.5.172         failed to recover cluster.
cinder/0*                    active    idle   10.1.5.155         
glance-mysql-router/0*       active    idle   10.1.5.169         
glance-mysql/0*              active    idle   10.1.5.182         Primary
glance/0*                    active    idle   10.1.5.137         
horizon-mysql-router/0*      error     idle   10.1.5.168         hook failed: "start"
horizon-mysql/0*             blocked   idle   10.1.5.164         failed to recover cluster.
horizon/0*                   active    idle   10.1.5.183         
keystone-mysql-router/0*     error     idle   10.1.5.166         hook failed: "start"
keystone-mysql/0*            blocked   idle   10.1.5.140         failed to recover cluster.
keystone/0*                  active    idle   10.1.5.165         
neutron-mysql-router/0*      active    idle   10.1.5.160         
neutron-mysql/0*             active    idle   10.1.5.142         Primary
neutron/0*                   active    idle   10.1.5.138         
nova-api-mysql-router/0*     error     idle   10.1.5.144         hook failed: "start"
nova-cell-mysql-router/0*    error     idle   10.1.5.133         hook failed: "start"
nova-mysql-router/0*         error     idle   10.1.5.151         hook failed: "start"
nova-mysql/0*                blocked   idle   10.1.5.186         failed to recover cluster.
nova/0*                      blocked   idle   10.1.5.178         (container:nova-api) healthcheck failed: online
ovn-central/0*               active    idle   10.1.5.141         
ovn-relay/0*                 active    idle   10.1.5.170         
placement-mysql-router/0*    active    idle   10.1.5.162         
placement-mysql/0*           active    idle   10.1.5.139         Primary
placement/0*                 waiting   idle   10.1.5.130         (workload) Not all relations are ready
rabbitmq/0*                  active    idle   10.1.5.171         
traefik/0*                   active    idle   10.1.5.145
2 Likes
3

Yeah same problem here. It seems to be related to mysql not coming back up… really unfortunate to see that this problem existed for some time now -.-

1 Like
4

Hello, I’m new here, and also having the the same problem described. After reboot, microstack doesn’t work. Does anyone figure out a fix or workaround? Thank you

5

Hello again, I found this today: Bug #2042351 “after reboot all mysql-routers report error” : Bugs : OpenStack Snap Noticed the same behaviour: some pods don’t start, with mysql connection error. After run this command and wait for about 10min, I was able to access the dashboard with my demo user. Hope it helps

kubectl -n openstack rollout restart statefulset/mysql

6

Hello, apparently this doesn’t solves this issue. Yesterday probably was a little bit of coincidence that things start working again…