Critical error during OVN migration

charm
,
#1

Following the documentation I encounter an unexpected error during database migration;

$ juju run-action --wait neutron-api-plugin-ovn/0 migrate-ovn-db i-really-mean-it=true
...
Stdout: "migrate-ovn-db: OUTPUT FROM SYNC ON STDOUT:\n2020-11-25 11:06:17.760
  413761 INFO neutron.cmd.ovn.neutron_ovn_db_sync_util [-] Started Neutron OVN
  db sync\e[00m\n
...
  11:06:57.026 413761 WARNING neutron.db.ovn_revision_numbers_db [req-91be95af-c8c7-41cc-a372-5cd7a0241373
  - - - - -] No revision row found for e50ff198-0fb5-4fd4-9b6c-d2f4b0a3f575 (type:
  networks) when bumping the revision number. Creating one.\e[00m\n2020-11-25
  11:06:57.227 413761 INFO neutron.db.ovn_revision_numbers_db [req-91be95af-c8c7-41cc-a372-5cd7a0241373
  - - - - -] Successfully bumped revision number for resource e50ff198-0fb5-4fd4-9b6c-d2f4b0a3f575
  (type: networks) to 10\e[00m\n2020-11-25 11:06:57.565 413761 CRITICAL neutron_ovn_db_sync_util
  [req-91be95af-c8c7-41cc-a372-5cd7a0241373 - - - - -] Unhandled error: neutron_lib.exceptions.IpAddressGenerationFailure:
  No more IP addresses available on network e50ff198-0fb5-4fd4-9b6c-d2f4b0a3f575.\n2020-11-25
  11:06:57.565 413761 ERROR neutron_ovn_db_sync_util Traceback (most recent call
  last):\n
...

I figure something strange with the configuration of this network e50ff198-0fb5-4fd4-9b6c-d2f4b0a3f575, but it results in a partial migration so need to unwind something to fix it.
The neutron-api is paused at this point and in the config, manage-neutron-plugin-legacy-mode=false. Can I safely resume the neutron-api to diagnose the network?
Many thanks in advance for any clue!

#2

It looks like this issue is being addressed in LP #1905554.

#3

Thanks. My report. Actually probably not a charm bug. The data is migrated by neutron_ovn_db_sync_util and it seems quite fragile. If there were instructions to set e.g. firewall-driver=openvswitch I missed them. A small amount of data might have been messed, but actually mostly the migration is working.

I do have a couple of hypervisors on which ovn-chassis is apparently missing neutron-ovn-metadata-agent. It is not installed. All others seem fine, e.g

nova-compute/14            active    idle   48       192.168.200.220           Unit is ready
  nova-compute-syslog/5    active    idle            192.168.200.220           Unit is ready
  ntp/190                  active    idle            192.168.200.220  123/udp  chrony: Ready
  ovn-chassis/0            blocked   idle            192.168.200.220           Services not running that should be: neutron-ovn-metadata-agent
nova-compute/16            active    idle   50       192.168.200.47            Unit is ready
  nova-compute-syslog/6    active    idle            192.168.200.47            Unit is ready
  ntp/191*                 active    idle            192.168.200.47   123/udp  chrony: Ready
  ovn-chassis/3            active    idle            192.168.200.47            Unit is ready

On which…

(osc) $ juju ssh ovn-chassis/0 -- dpkg -l neutron-ovn-metadata-agent
dpkg-query: no packages found matching neutron-ovn-metadata-agent
Connection to 192.168.200.220 closed.
(osc) $ juju ssh ovn-chassis/3 -- dpkg -l neutron-ovn-metadata-agent
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                       Version           Architecture Description
+++-==========================-=================-============-=======================================================================
ii  neutron-ovn-metadata-agent 2:16.2.0-0ubuntu1 all          Neutron is a virtual network service for Openstack - OVN metadata agent
Connection to 192.168.200.47 closed.

Actually I did get a bunch of metadata port errors during migration so I expect somehow this might be a hangover of a previously broken hypervisor?

Can I recover it without rebuilding?

#4

Great to hear that you were able to move forward with the migration. The issue you mention here with some of the ovn-chassis units not having installed the neutron-ovn-metadata-agent package is interesting.

Installing it is decided at runtime depending on whether ovn-chassis has a relation to nova-compute or not to support using it with CMS’es other than OpenStack.

I assume you deployed the ovn-chassis charm with the new-units-paused configuration option set to support the migration, and there may be a race condition which results in the package not being installed when you resume the charm from its pause.

Would you be able to raise a bug targeting the charm-layer-ovn project on Launchpad and include the complete charm log from the affected unit(s).

1 Like
#5

Very sorry I don’t have logs now, I had to press on and I have re-deployed the broken compute units. Thanks anyway.

#6

No worries, thank you for reporting back. We will try to recreate the situation in our lab.

A possible way to salvage the situation could be to force a full hook execution. You can do that with this command: juju run --unit ovn-chassis/0 hooks/config-changed