Charm - mysql-innodb-cluster stuck waiting machine - openstack installation step

I install mysql-innodb-cluster for openstack from this tutorial https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/install-openstack.html

start from MAAS adn JUJU installation, to ceph-nova smooth. But I got stuck when install mysql-innodb-cluster

Please advise, thank you!

@billy-olsen any insight here? Looks like possibly a lxd issue?

Hmm, I see container started but state pending for the juju machine agent, which to me suggests that there may be some networking challenges for the juju machine agent to reach back to the juju controller.

You can juju ssh 0 to log into machine 0 and then lxc list to view the running containers. Machine 0 should have a container named juju-05c63e-0-lxd-2. You’ll want to check that it has an IP address assigned - it shows in the lxc listing, e.g.:

$ lxc list
+---------------------+---------+---------------------+----------------------------------------------+-----------+-----------+
|        NAME         |  STATE  |        IPV4         |                     IPV6                     |   TYPE    | SNAPSHOTS |
+---------------------+---------+---------------------+----------------------------------------------+-----------+-----------+
| juju-40786f-0       | RUNNING | 10.167.81.32 (eth0) | fd42:fc94:4e5c:5216:216:3eff:fe66:d07 (eth0) | CONTAINER | 0         |
+---------------------+---------+---------------------+----------------------------------------------+-----------+-----------+

Next, I’d hop into the lxd container and check that it can reach the juju controller and check the machine agent logs. To get into the container you can run lxc exec juju-05c63e-0-lxd-2 /bin/bash to get a shell and examine logs, networking etc. Logs should be available in /var/log/juju.

This is where I’d start to look to get an idea of what may be causing some troubles.

Hi @hpidcock Thank you for interest Hello @billy-olsen , here is the sample detail: lxc list To start your first container, try: lxc launch ubuntu:22.04 Or for a virtual machine: lxc launch ubuntu:22.04 --vm

±--------------------±--------±-------------------±-----±----------±----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | ±--------------------±--------±-------------------±-----±----------±----------+ | juju-05c63e-0-lxd-3 | RUNNING | 11.0.12.216 (eth0) | | CONTAINER | 0 | ±--------------------±--------±-------------------±-----±----------±----------+

For the log in juju ssh 0 (/var/log/juju/machine-0.log) on the attachement(new user cant log attachment):

2023-07-05 06:00:36 INFO juju.worker.lxdprovisioner provisioner_task.go:1486 started machine 0/lxd/2 as instance juju-05c63e-0-lxd-2 with hardware “availability-zone=default”, network config [], volumes [], volume attachments map[], subnets to zones [], lxd profiles [] 2023-07-05 06:00:36 INFO juju.worker.instancemutater.container mutater.go:245 no changes necessary to machine-0/lxd/2 lxd profiles ([default]) 2023-07-05 06:22:52 INFO juju.worker.diskmanager diskmanager.go:67 block devices changed: []storage.BlockDevice{storage.BlockDevice{DeviceName:“loop0”, DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x3f, FilesystemType:“squashfs”, InUse:true, MountPoint:"/snap/core20/1852", SerialId:""}, storage.BlockDevice{DeviceName:“loop1”, DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x6f, FilesystemType:“squashfs”, InUse:true, MountPoint:"/snap/lxd/24322", SerialId:""}, storage.BlockDevice{DeviceName:“loop2”, DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x31, FilesystemType:“squashfs”, InUse:true, MountPoint:"/snap/snapd/18596", SerialId:""}, storage.BlockDevice{DeviceName:“loop3”, DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x35, FilesystemType:“squashfs”, InUse:true, MountPoint:"/snap/snapd/19457", SerialId:""}, storage.BlockDevice{DeviceName:“sda”, DeviceLinks:[]string{"/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0"}, Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:“scsi@32:0.0.0”, Size:0x28000, FilesystemType:"", InUse:true, MountPoint:"", SerialId:""}, storage.BlockDevice{DeviceName:“sda1”, DeviceLinks:[]string{"/dev/disk/by-partuuid/d39c27e7-5af7-4810-9487-02a388b3936e", “/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0-part1”}, Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:“scsi@32:0.0.0”, Size:0x1, FilesystemType:"", InUse:false, MountPoint:"", SerialId:""}, storage.BlockDevice{DeviceName:“sda2”, DeviceLinks:[]string{"/dev/disk/by-dname/sda-part2", “/dev/disk/by-label/root”, “/dev/disk/by-partuuid/f1f054a5-6cd6-4fec-a8ae-c0df28d62621”, “/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0-part2”, “/dev/disk/by-uuid/2bf8149b-e17c-490b-905e-1458a0e83f3b”}, Label:“root”, UUID:“2bf8149b-e17c-490b-905e-1458a0e83f3b”, HardwareId:"", WWN:"", BusAddress:“scsi@32:0.0.0”, Size:0x27ffd, FilesystemType:“ext4”, InUse:true, MountPoint:"/", SerialId:""}, storage.BlockDevice{DeviceName:“sdb”, DeviceLinks:[]string{"/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:1:0"}, Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:“scsi@32:0.1.0”, Size:0x19000, FilesystemType:"", InUse:false, MountPoint:"", SerialId:""}} 2023-07-05 06:24:22 INFO juju.worker.diskmanager diskmanager.go:67 block devices changed: []storage.BlockDevice{storage.BlockDevice{DeviceName:“loop0”, DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x3f, FilesystemType:“squashfs”, InUse:true, MountPoint:"/snap/core20/1852", SerialId:""}, storage.BlockDevice{DeviceName:“loop1”, DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x6f, FilesystemType:“squashfs”, InUse:true, MountPoint:"/snap/lxd/24322", SerialId:""}, storage.BlockDevice{DeviceName:“loop2”, DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x31, FilesystemType:“squashfs”, InUse:true, MountPoint:"/snap/snapd/18596", SerialId:""}, storage.BlockDevice{DeviceName:“loop3”, DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x35, FilesystemType:“squashfs”, InUse:true, MountPoint:"/snap/snapd/19457", SerialId:""}, storage.BlockDevice{DeviceName:“loop4”, DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x3f, FilesystemType:“squashfs”, InUse:true, MountPoint:"/snap/core20/1974", SerialId:""}, storage.BlockDevice{DeviceName:“sda”, DeviceLinks:[]string{"/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0"}, Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:“scsi@32:0.0.0”, Size:0x28000, FilesystemType:"", InUse:true, MountPoint:"", SerialId:""}, storage.BlockDevice{DeviceName:“sda1”, DeviceLinks:[]string{"/dev/disk/by-partuuid/d39c27e7-5af7-4810-9487-02a388b3936e", “/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0-part1”}, Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:“scsi@32:0.0.0”, Size:0x1, FilesystemType:"", InUse:false, MountPoint:"", SerialId:""}, storage.BlockDevice{DeviceName:“sda2”, DeviceLinks:[]string{"/dev/disk/by-dname/sda-part2", “/dev/disk/by-label/root”, “/dev/disk/by-partuuid/f1f054a5-6cd6-4fec-a8ae-c0df28d62621”, “/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0-part2”, “/dev/disk/by-uuid/2bf8149b-e17c-490b-905e-1458a0e83f3b”}, Label:“root”, UUID:“2bf8149b-e17c-490b-905e-1458a0e83f3b”, HardwareId:"", WWN:"", BusAddress:“scsi@32:0.0.0”, Size:0x27ffd, FilesystemType:“ext4”, InUse:true, MountPoint:"/", SerialId:""}, storage.BlockDevice{DeviceName:“sdb”, DeviceLinks:[]string{"/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:1:0"}, Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:“scsi@32:0.1.0”, Size:0x19000, FilesystemType:"", InUse:false, MountPoint:"", SerialId:""}} 2023-07-05 06:41:27 INFO juju.worker.lxdprovisioner provisioner_task.go:629 stopping known instances [juju-05c63e-0-lxd-2] 2023-07-05 06:41:27 INFO juju.container.broker.lxd lxd-broker.go:142 stopping lxd container for instance: juju-05c63e-0-lxd-2 2023-07-05 06:41:30 INFO juju.container.broker.lxd broker.go:146 released all addresses for container “0/lxd/2” 2023-07-05 06:41:30 INFO juju.worker.lxdprovisioner provisioner_task.go:645 removing dead machine “0/lxd/2” 2023-07-05 06:43:22 INFO juju.worker.lxdprovisioner provisioner_task.go:504 found machine pending provisioning id:0/lxd/3, details:0/lxd/3 2023-07-05 06:43:25 WARNING juju.container.broker broker.go:76 no search domains supplied by provider, using host’s search domains. 2023-07-05 06:43:25 WARNING juju.container.broker broker.go:84 incomplete DNS config found, discovering host’s DNS config 2023-07-05 06:43:25 INFO juju.cloudconfig userdatacfg_unix.go:613 Fetching agent: curl -sSf --connect-timeout 20 --noproxy “*” --insecure -o $bin/tools.tar.gz <[https://11.0.12.201:17070/model/8e869d67-0a1f-4fde-8778-added705c63e/tools/2.9.43-ubuntu-amd64]> 2023-07-05 06:43:25 INFO juju.container.lxd container.go:256 starting new container “juju-05c63e-0-lxd-3” (image “ubuntu-22.04-server-cloudimg-amd64-lxd.tar.xz”) 2023-07-05 06:43:40 INFO juju.worker.lxdprovisioner provisioner_task.go:1486 started machine 0/lxd/3 as instance juju-05c63e-0-lxd-3 with hardware “availability-zone=default”, network config [], volumes [], volume attachments map[], subnets to zones [], lxd profiles [] 2023-07-05 06:43:41 INFO juju.worker.instancemutater.container mutater.go:245 no changes necessary to machine-0/lxd/3 lxd profiles ([default]) 2023-07-09 15:00:58 INFO juju.cmd supercommand.go:56 running jujud [2.9.43 3cb3f8beac4a0b05e10bdfb8014f5666118a269d gc go1.20.4] 2023-07-09 15:00:58 DEBUG juju.cmd supercommand.go:57 args: []string{"/var/lib/juju/tools/machine-0/jujud", “machine”, “–data-dir”, “/var/lib/juju”, “–machine-id”, “0”, “–debug”} 2023-07-09 15:00:58 DEBUG juju.utils gomaxprocs.go:24 setting GOMAXPROCS to 4 2023-07-09 15:00:58 DEBUG juju.agent agent.go:603 read agent config, format “2.0” 2023-07-09 15:00:58 INFO juju.agent.setup agentconf.go:128 setting logging config to “=INFO” 2023-07-09 15:00:58 INFO juju.worker.upgradesteps worker.go:60 upgrade steps for 2.9.43 have already been run. 2023-07-09 15:00:58 INFO juju.api apiclient.go:687 connection established to “wss://11.0.12.201:17070/model/8e869d67-0a1f-4fde-8778-added705c63e/api” 2023-07-09 15:00:58 INFO juju.worker.apicaller connect.go:163 [8e869d] “machine-0” successfully connected to “11.0.12.201:17070” 2023-07-09 15:00:58 INFO juju.api apiclient.go:687 connection established to “wss://11.0.12.201:17070/model/8e869d67-0a1f-4fde-8778-added705c63e/api” 2023-07-09 15:00:58 INFO juju.worker.apicaller connect.go:163 [8e869d] “machine-0” successfully connected to “11.0.12.201:17070” 2023-07-09 15:00:58 INFO juju.worker.upgrader upgrader.go:241 desired agent binary version: 2.9.43 2023-07-09 15:00:58 INFO juju.worker.deployer nested.go:159 new context: units “ceph-osd/0”, stopped “” 2023-07-09 15:00:59 INFO juju.worker.deployer unit_agent.go:125 creating new agent config for “ceph-osd/0” 2023-07-09 15:00:59 INFO juju.worker.deployer nested.go:366 starting workers for “ceph-osd/0” 2023-07-09 15:00:59 INFO juju.worker.deployer deployer.go:161 checking unit “ceph-osd/0” 2023-07-09 15:00:59 INFO juju.worker.migrationminion worker.go:142 migration phase is now: NONE 2023-07-09 15:00:59 INFO juju.worker.logger logger.go:120 logger worker started 2023-07-09 15:00:59 INFO juju.worker.deployer deployer.go:161 checking unit “ceph-osd/0” 2023-07-09 15:00:59 INFO juju.api apiclient.go:687 connection established to “wss://11.0.12.201:17070/model/8e869d67-0a1f-4fde-8778-added705c63e/api” 2023-07-09 15:00:59 INFO juju.worker.upgradeseries worker.go:162 no series upgrade lock present 2023-07-09 15:00:59 INFO juju.worker.machiner machiner.go:163 setting addresses for “machine-0” to [local-machine:127.0.0.1 public:11.0.12.202 local-machine:::1] 2023-07-09 15:00:59 INFO juju.packaging.manager run.go:88 Running: snap info lxd 2023-07-09 15:00:59 INFO juju.worker.machiner machiner.go:113 “machine-0” started 2023-07-09 15:00:59 INFO juju.worker.leadership tracker.go:217 ceph-osd leadership for ceph-osd/0 denied 2023-07-09 15:00:59 INFO juju.worker.authenticationworker worker.go:101 “machine-0” key updater worker started 2023-07-09 15:00:59 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-ceph-osd-0 2023-07-09 15:00:59 INFO juju.agent.tools symlinks.go:40 was a symlink, now looking at /var/lib/juju/tools/2.9.43-ubuntu-amd64 2023-07-09 15:00:59 INFO juju.worker.diskmanager diskmanager.go:67 block devices changed: []storage.BlockDevice{storage.BlockDevice{DeviceName:“loop0”, DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x3f, FilesystemType:“squashfs”, InUse:true, MountPoint:"/snap/core20/1974", SerialId:""}, storage.BlockDevice{DeviceName:“loop1”, DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x3f, FilesystemType:“squashfs”, InUse:true, MountPoint:"/snap/core20/1852", SerialId:""}, storage.BlockDevice{DeviceName:“loop2”, DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x6f, FilesystemType:“squashfs”, InUse:true, MountPoint:"/snap/lxd/24322", SerialId:""}, storage.BlockDevice{DeviceName:“loop3”, DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x31, FilesystemType:“squashfs”, InUse:true, MountPoint:"/snap/snapd/18596", SerialId:""}, storage.BlockDevice{DeviceName:“loop4”, DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x35, FilesystemType:“squashfs”, InUse:true, MountPoint:"/snap/snapd/19457", SerialId:""}, storage.BlockDevice{DeviceName:“sda”, DeviceLinks:[]string{"/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0"}, Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:“scsi@32:0.0.0”, Size:0x28000, FilesystemType:"", InUse:true, MountPoint:"", SerialId:""}, storage.BlockDevice{DeviceName:“sda1”, DeviceLinks:[]string{"/dev/disk/by-partuuid/d39c27e7-5af7-4810-9487-02a388b3936e", “/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0-part1”}, Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:“scsi@32:0.0.0”, Size:0x1, FilesystemType:"", InUse:false, MountPoint:"", SerialId:""}, storage.BlockDevice{DeviceName:“sda2”, DeviceLinks:[]string{"/dev/disk/by-dname/sda-part2", “/dev/disk/by-label/root”, “/dev/disk/by-partuuid/f1f054a5-6cd6-4fec-a8ae-c0df28d62621”, “/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:0:0-part2”, “/dev/disk/by-uuid/2bf8149b-e17c-490b-905e-1458a0e83f3b”}, Label:“root”, UUID:“2bf8149b-e17c-490b-905e-1458a0e83f3b”, HardwareId:"", WWN:"", BusAddress:“scsi@32:0.0.0”, Size:0x27ffd, FilesystemType:“ext4”, InUse:true, MountPoint:"/", SerialId:""}, storage.BlockDevice{DeviceName:“sdb”, DeviceLinks:[]string{"/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:1:0"}, Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:“scsi@32:0.1.0”, Size:0x19000, FilesystemType:"", InUse:false, MountPoint:"", SerialId:""}} 2023-07-09 15:01:01 INFO juju.container.lxd initialisation_linux.go:295 LXD snap is already installed (channel: 5.0/stable/ubuntu-22.04); skipping package installation 2023-07-09 15:01:10 INFO juju.worker.instancemutater.container mutater.go:245 no changes necessary to machine-0/lxd/3 lxd profiles ([default]) 2023-07-09 15:01:11 INFO juju.worker.instancemutater.container mutater.go:245 no changes necessary to machine-0/lxd/3 lxd profiles ([default]) 2023-07-09 15:01:13 INFO juju.worker.lxdprovisioner provisioner_task.go:203 entering provisioner task loop; using provisioner pool with 4 workers 2023-07-09 15:01:13 INFO juju.worker.lxdprovisioner provisioner_task.go:518 machine 0/lxd/3 already started as instance “juju-05c63e-0-lxd-3”

Thank you for the support discussion