The 10.20.20.0/24 network is no longer accessible from the host machine, essentially breaking our cluster. The br-ex interface is still up as 10.20.20.1, bu no arp requests are answered. This seems to have occurred after (failed) attempts to add a second compute node to the cluster. How can we go about debugging and resolving this?
Hello apb. Thank you for the question!
You might take a look at the microstack config, to see if it is still using the network that you expect it to be using:
sudo snap get microstack config.network
If that looks like what you expect, please provide some more details about what went wrong with the attempt to connect the compute node.
Note that you can restart microstack with sudo snap restart microstack, and get an overview of running services, along with error messages, by running systemctl status snap.microstack.*
Thank you for the reply. The settings seem correct.
One of the main instances we’re running is configured with a floating IP that we have been accessing (and forwarding data to w/ socat) from the host:
external 10.20.20.8
The issue is that we cannot reach this IP, or any on the 10.20.20.0/24 subnet, from the host any longer.
The security group on the port & instance itself is the same, with the following rules:
default
ALLOW IPv4 22/tcp from 0.0.0.0/0
ALLOW IPv4 icmp from 0.0.0.0/0
ALLOW IPv4 to 0.0.0.0/0
ALLOW IPv4 tcp from 0.0.0.0/0
ALLOW IPv6 from default
ALLOW IPv6 to ::/0
ALLOW IPv4 udp from 0.0.0.0/0
ALLOW IPv4 from default
It appears that the correct interface is being used for the connections:
# ip r get 10.20.20.8
10.20.20.8 dev br-ex src 10.20.20.1
cache
This nat rule is set:
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 10.20.20.0/24 !10.20.20.0/24
And ip forwarding is enabled:
# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
And this is the output of microstack.ovs-vsctl show:
8594a53a-434f-4f0c-8e1e-a9924745d7f9
Bridge br-ex
datapath_type: system
Port patch-provnet-c93a635f-27a8-447a-93e3-1906674adac7-to-br-int
Interface patch-provnet-c93a635f-27a8-447a-93e3-1906674adac7-to-br-int
type: patch
options: {peer=patch-br-int-to-provnet-c93a635f-27a8-447a-93e3-1906674adac7}
Port br-ex
Interface br-ex
type: internal
Bridge br-int
datapath_type: system
Port patch-br-int-to-provnet-c93a635f-27a8-447a-93e3-1906674adac7
Interface patch-br-int-to-provnet-c93a635f-27a8-447a-93e3-1906674adac7
type: patch
options: {peer=patch-provnet-c93a635f-27a8-447a-93e3-1906674adac7-to-br-int}
Port tap16b52ffc-e0
Interface tap16b52ffc-e0
error: "could not open network device tap16b52ffc-e0 (No such device)"
Port tap20634d8b-88
Interface tap20634d8b-88
Port tap77ea4fe4-8f
Interface tap77ea4fe4-8f
Port snooper0
Interface snooper0
Port tap6606383d-70
Interface tap6606383d-70
Port br-int
Interface br-int
type: internal
Port ovn-ddda98-0
Interface ovn-ddda98-0
type: geneve
options: {csum="true", key=flow, remote_ip="X.X.X.X}
bfd_status: {diagnostic="No Diagnostic", flap_count="0", forwarding="false", remote_diagnostic="No Diagnostic", remote_state=down, state=down}
Port tapd1bce8eb-b0
Interface tapd1bce8eb-b0
Port tapfd3b2c8c-f0
Interface tapfd3b2c8c-f0
ovs_version: "2.13.0"
The snooper0 port is one we added to be able to tcpdump on the internal interfaces during debugging, so you can probably ignore that. The ovn-ddda98-0 port was added when we attempted to add a second compute node to the cluster. We have since removed the node from the cluster (microstack.openstack compute service...) but this port remains.
Any help will be immensely appreciated.
Also, here is tcpdump output of br-ex while attempting to ssh to the host - it’s just unanswered arp requests.
# sudo tcpdump -enX -i br-ex
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-ex, link-type EN10MB (Ethernet), capture size 262144 bytes
22:10:09.911119 72:c6:81:07:7e:41 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.20.20.8 tell 10.20.20.1, length 28
0x0000: 0001 0800 0604 0001 72c6 8107 7e41 0a14 ........r...~A..
0x0010: 1401 0000 0000 0000 0a14 1408 ............
22:10:10.925832 72:c6:81:07:7e:41 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.20.20.8 tell 10.20.20.1, length 28
0x0000: 0001 0800 0604 0001 72c6 8107 7e41 0a14 ........r...~A..
0x0010: 1401 0000 0000 0000 0a14 1408 ............
22:10:11.949823 72:c6:81:07:7e:41 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.20.20.8 tell 10.20.20.1, length 28
0x0000: 0001 0800 0604 0001 72c6 8107 7e41 0a14 ........r...~A..
0x0010: 1401 0000 0000 0000 0a14 1408 ............
22:10:12.973888 72:c6:81:07:7e:41 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.20.20.8 tell 10.20.20.1, length 28
0x0000: 0001 0800 0604 0001 72c6 8107 7e41 0a14 ........r...~A..
0x0010: 1401 0000 0000 0000 0a14 1408 ............
22:10:13.997815 72:c6:81:07:7e:41 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.20.20.8 tell 10.20.20.1, length 28
0x0000: 0001 0800 0604 0001 72c6 8107 7e41 0a14 ........r...~A..
0x0010: 1401 0000 0000 0000 0a14 1408 ............
22:10:15.021826 72:c6:81:07:7e:41 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.20.20.8 tell 10.20.20.1, length 28
0x0000: 0001 0800 0604 0001 72c6 8107 7e41 0a14 ........r...~A..
0x0010: 1401 0000 0000 0000 0a14 1408 ............Well, after a full day shot on debugging this, it is finally fixed.
The issue preventing me from pinging the VM with an interface in 10.20.20.0/24 was actually just a matter of the interface name changing, causing the netplan config to break. After fixing that I again hit the issue that I had begun with (not mentioned in the OP) - the VMs could not access the internet.
After tcpdump’ing the bridges it was clear that things weren’t being bridged properly in some way. I could communicate with hosts assigned interfaces in the 10.20.20.0/24 subnet directly, but the router interface for this external subnet did not seem to pass packets along to the host. The host could ping 10.20.20.1 but not 10.20.20.15 (the interface on the router) and vice versa for the VMs.
Deleting the host that had failed to be added to the cluster from the service list still left references to it in many places, which maybe is expected. I deleted it from a few tables in nova-related databases, and tried to delete the network agents related to it but was unable to. After looking for more networking-related places where references to the failed node could be hiding, I checked the chassis:
# microstack.ovn-sbctl show
Chassis "ae98e790-9a1a-4820-ab0b-4f8152568af4"
hostname: WORKING_HOST
Encap geneve
ip: "10.20.20.1"
options: {csum="true"}
Port_Binding "79f6155b-d8cb-4b21-ac53-3ef51cef1ac3"
Port_Binding "ce06050d-7ccf-40e9-acc5-5513cdc5c34e"
Port_Binding "77ea4fe4-8f50-4fa0-94cf-40bde2165057"
Port_Binding "6b229da2-5097-42ed-babd-c4449da3a1f4"
Port_Binding "d1bce8eb-b012-4859-9579-d6900ae62f8e"
Port_Binding "dfd10334-2b79-4ea5-bdbe-97864fa3baa0"
Chassis "ddda9808-8852-4b36-a978-0015a552bdd6"
hostname: FAILED_HOST
Encap geneve
ip: "X.X.X.X"
options: {csum="true"}
Deleting the remnant chassis completed successfully:
# microstack.ovn-sbctl chassis-del ddda9808-8852-4b36-a978-0015a552bdd6
And finally, from a VM:
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=1.42 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=1.44 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=1.37 ms
Which brings me to my next question - is adding a second compute node to a cluster managed by Microstack reliable and/or recommended? It had failed for me due to a bug that I’ve since reported, but even after a workaround it continued to fail for reasons I did not bother diving into. To make matters worse, the attempted installation caused the networking issues above.
Hi apb. Thank you for the extensive debugging info.
MicroStack is still in beta, and the clustering code is relatively new, which unfortunately means that you may run into issues like this. I’ll link the folks currently working on MicroStack to this thread, as your detective work may help them nail down some of the kinks in MicroStack networking.
1 Like