OpenStack Trouble - MySQL Router not yet bootstrapped (keystone-mysql-router)

First-time juju user, and thought I might attempt a Charmed Openstack install - because throwing myself in the deep-end is always fun (and having an Openstack private cloud would be most useful right now). For this case, I’ve tried to follow the guide: https://docs.openstack.org/charm-guide/latest/index.html … as close as possible, although there is some variation in operating environment (later …)

For now, I’ve just completed the RabbitMQ step (as explained here: https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/install-openstack.html) which is a stop-and-check point. When checking, the following status is shown:

As can be seen, I have a I have a “MySQL Router not yet bootstrapped” issue for the keystone-mysql-router app.

I’m not really sure how to go about solving the issue. I’ve had a hunt around, and it was suggested elsewhere to consider logging onto each node and manually updating apt packages. I have attempted this, and for nodes 0,1,2 updates went through without a problem (no change in status however), but for node 3, I have the following:

dpkg: dependency problems prevent configuration of mysql-server:
 mysql-server depends on mysql-server-8.0; however:
  Package mysql-server-8.0 is not configured yet.

and

dpkg: error processing package mysql-server (--configure):
 dependency problems - leaving unconfigured
Setting up libcgi-pm-perl (4.54-1) ...
No apport report written because the error message indicates its a followup error from a previous failure.

I’m only assuming this is related to the problem - it could be something else.

I’m not sure the best way to attack this one, as I don’t want to break Juju’s ability to manage things, and I still have a lot to learn about how to use it. Any assistance on how best to diagnose the root-fault would be most appreciated. If I can be pointed in a direction to solve it, that would be a bonus …

Environment config

  • This install is being attempted in a nested virtual environment.
  • Each worker node is configured with 3x80Gb disks, 16Gb of RAM and 8-cpu cores.
  • Along with the 4 worker nodes, I have a 5th which is where MAAS and JuJu controllers reside (seperate VM’s). MAAS is providing network services (DHCP and DNS)
  • Rather than let MAAS allocate DHCP for the OVN-bridges (br-ex), I decided to set static IP’s. I haven’t encountered any issues up until this point, so assume that’s not going to impact anything.
  • Because this is a nested environment, LXD’s will not work (a change to app armor after focal fossa?). Consequently, any step such as
juju deploy -n 3 --to lxd:0,lxd:1,lxd:2

was changed to:

juju deploy -n 3 --to 0,1,2

So, I got a little impatient, and decide to juju remove-application both keystone-mysql-router and keystone apps.

Once everything completed, I went to re-deploy keystone, but failed to specify the channel (smh). This resulted in a failed “yoga” install. I then had to remove-application --force to get rid of it.

Finally, I re-deployed the correct release for this case (2023.2/stable), but once again see a “Allowed_units list provided but this unit not present”. Doing some searching, I find bug Bug #2064127 “Inconsistent relation data when leadership changes...” : Bugs : MySQL InnoDB Cluster Charm

Not sure if it’s exactly the same issue, but will continue trying to get my head around it and see if anything can be transposed to this issue.

<tap, tap> … “Is this thing on?”

sigh

Since this is a non-production envrionment, and Juju 3.6.0 has now been released, I’ve decided to blow everything away, bootstrap a new controller, recommission machines in MAAS and start over once again. Unfortunately, I not only have the same keystone related issue come up, I seem to have picked up a couple of new ones as well (neutron).

I wouldn’t expect this to a common problem, and certainly searching through older posts isn’t giving me much to go on, so there must be a fundemental step or something key to my environment that’s tripping things up - obscure config error, a broken charm version, hypervisor, not using LXD isolation … no idea.

I have some logs this time, and will start going through them to see what I can see … but I’m kind of stumbling in the dark to be honest.

Machine 0 logs: unit-keystone-0.log.pdf (89.8 KB) unit-keystone-mysql-router-0.log.pdf (75.6 KB) unit-mysql-innodb-cluster-0.log.pdf (487.8 KB)