Any one had this error before?
Error creating token: Post https://127.0.0.1:8200/v1/auth/token/create: dial tcp 127.0.0.1:8200: connect: connection refused
It seems that Vault is bound to the loopback address and I cannot find a YAML or config file to modify the IP address it is bound to. Any ideas is would be appriciated.
That interface should be being listened to by the vault service if it is running.
Check the status of the vault service and check logs for why that service may have failed to start.
systemctl status vault
configuration is in /var/snap/vault/common/vault.hcl.
It should be listening on all ipv4 and ipv6 interfaces (including localhost/127.0.0.1) on port 8200:
address = "[::]:8200"
-Drew
Thanks Drew,
I checked the log files and Vault status and all looks fine. I unsealed Vault again and the vault is running well. However, I get hook failed: "ovsdb-relation-changed"
The main error I get now is hook failed: "ovsdb-relation-changed"
How do I clear the hook failed error?
Tabi
I checked the logs on the nodes running ovsdb-server and found same error:
ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.15.0
|jsonrpc|WARN|unix#0: receive error: Connection reset by peer
After restarting the ovsdb-server service, still getting same error and the juju status on the application is still the same: hook failed: "ovsdb-relation-changed"
I am running the latest Openstack on Ubuntu focal.
The juju debug logs for the ovn-chassis units should show the exception trail and command that failed causing the hook failure. Those logs would be in /var/log/juju/unit-ovn-chassis-*.log on the machines, or queryable with juju debug-log -i ovn-chassis/4.
juju resolved ovn-chassis/4 would try to rerun the hook that failed once you’ve resolved the underlying issue, or if your “juju model-config automatically-retry-hooks” is set to “true”, this retry should happen automatically every minute or so.
This can happen when the interface assigned to ovn-chassis is incorrect. What is the output to:
juju config ovn-chassis bridge-interface-mappings
Then compare that to the available interfaces, for each unit:
juju ssh ovn-chassis/X ip a
If that looks good then try, for each unit:
juju resolve --no-retry ovn-chassis/XThanks Peter and Drew.
Running sudo ovs-vsctl show on the hosts, I get this:
7b714304-d820-4200-9ac6-da03b9961ee9
Manager “ptcp:6640:127.0.0.1”
Bridge br-ex
fail_mode: standalone
datapath_type: system
Port eno2
Interface eno2
type: system
error: "could not open network device eno2 (No such device)"
Port br-ex
Interface br-ex
type: internal
Bridge br-int
fail_mode: secure
datapath_type: system
Port br-int
Interface br-int
type: internal
ovs_version: “2.15.0”
It seems like the problem is with the interface eno2. Does not exist.
I ran the commands your suggested and got the following:
admin@north:~$ juju ssh ovn-chassis/2 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens20f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-ens20f0 state UP group default qlen 1000
link/ether 54:ab:3a:70:d8:77 brd ff:ff:ff:ff:ff:ff
3: ens20f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 54:ab:3a:70:d8:78 brd ff:ff:ff:ff:ff:ff
inet 10.101.15.81/24 brd 10.101.15.255 scope global ens20f1
valid_lft forever preferred_lft forever
inet6 fe80::56ab:3aff:fe70:d878/64 scope link
valid_lft forever preferred_lft forever
4: lxdbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 00:16:3e:ac:6c:57 brd ff:ff:ff:ff:ff:ff
inet 10.188.123.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
7: br-ens20f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 42:55:c9:f4:a1:52 brd ff:ff:ff:ff:ff:ff
inet 192.168.7.18/24 brd 192.168.7.255 scope global br-ens20f0
valid_lft forever preferred_lft forever
inet6 fe80::b0b9:75ff:fe4e:7253/64 scope link
valid_lft forever preferred_lft forever
8: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 9e:36:3e:5e:72:19 brd ff:ff:ff:ff:ff:ff
11: 12lxd2-0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether 7a:47:12:2e:89:6a brd ff:ff:ff:ff:ff:ff link-netnsid 0
13: 12lxd4-0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether ca:e4:cc:58:74:de brd ff:ff:ff:ff:ff:ff link-netnsid 1
15: 12lxd5-0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether 42:55:c9:f4:a1:52 brd ff:ff:ff:ff:ff:ff link-netnsid 2
17: 12lxd6-0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether 92:60:79:f3:08:8f brd ff:ff:ff:ff:ff:ff link-netnsid 3
19: 12lxd3-0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether 6e:0d:0a:ce:01:8a brd ff:ff:ff:ff:ff:ff link-netnsid 4
21: 12lxd1-0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether 6a:d0:42:46:7c:bc brd ff:ff:ff:ff:ff:ff link-netnsid 5
23: 12lxd0-0@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether da:51:c3:cc:d7:c3 brd ff:ff:ff:ff:ff:ff link-netnsid 6
25: br-ex: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 26:ba:37:66:8e:4c brd ff:ff:ff:ff:ff:ff
26: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether ca:0f:7f:10:4a:b6 brd ff:ff:ff:ff:ff:ff
Connection to 10.101.15.81 closed.
========================
admin@north:~$ juju ssh ovn-chassis/3 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens20f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-ens20f0 state UP group default qlen 1000
link/ether 54:ab:3a:3d:2b:06 brd ff:ff:ff:ff:ff:ff
3: ens20f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 54:ab:3a:3d:2b:07 brd ff:ff:ff:ff:ff:ff
inet6 fe80::56ab:3aff:fe3d:2b07/64 scope link
valid_lft forever preferred_lft forever
4: lxdbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 00:16:3e:49:6e:c6 brd ff:ff:ff:ff:ff:ff
inet 10.186.47.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
7: br-ens20f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 54:ab:3a:3d:2b:06 brd ff:ff:ff:ff:ff:ff
inet 192.168.7.40/24 brd 192.168.7.255 scope global br-ens20f0
valid_lft forever preferred_lft forever
inet6 fe80::a8c7:4bff:fe62:c1b7/64 scope link
valid_lft forever preferred_lft forever
8: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether e6:01:75:b6:45:00 brd ff:ff:ff:ff:ff:ff
11: 13lxd2-0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether 72:0f:13:7c:8b:64 brd ff:ff:ff:ff:ff:ff link-netnsid 1
13: 13lxd3-0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether 82:3f:92:4d:92:61 brd ff:ff:ff:ff:ff:ff link-netnsid 0
15: 13lxd1-0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether 9e:ae:a2:96:a7:e5 brd ff:ff:ff:ff:ff:ff link-netnsid 2
17: 13lxd4-0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether ce:3e:ae:33:41:4f brd ff:ff:ff:ff:ff:ff link-netnsid 5
19: 13lxd5-0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether 6a:f4:61:ea:e1:17 brd ff:ff:ff:ff:ff:ff link-netnsid 4
21: 13lxd0-0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether 9a:75:20:30:0b:d2 brd ff:ff:ff:ff:ff:ff link-netnsid 3
23: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 36:09:eb:c4:f1:c1 brd ff:ff:ff:ff:ff:ff
24: br-ex: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 7a:5f:4d:df:85:4b brd ff:ff:ff:ff:ff:ff
Connection to 192.168.7.40 closed.
=======================================
admin@north:~$ juju ssh ovn-chassis/4 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens20f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-ens20f0 state UP group default qlen 1000
link/ether 54:ab:3a:70:d4:03 brd ff:ff:ff:ff:ff:ff
3: ens20f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 54:ab:3a:70:d4:04 brd ff:ff:ff:ff:ff:ff
inet6 fe80::56ab:3aff:fe70:d404/64 scope link
valid_lft forever preferred_lft forever
4: lxdbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 00:16:3e:ba:56:41 brd ff:ff:ff:ff:ff:ff
inet 10.61.223.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
7: br-ens20f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 54:ab:3a:70:d4:03 brd ff:ff:ff:ff:ff:ff
inet 192.168.7.41/24 brd 192.168.7.255 scope global br-ens20f0
valid_lft forever preferred_lft forever
inet6 fe80::f88a:c7ff:fe6b:8b5e/64 scope link
valid_lft forever preferred_lft forever
8: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 7e:e8:82:3d:79:bf brd ff:ff:ff:ff:ff:ff
11: 14lxd1-0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether 76:f9:76:b7:97:50 brd ff:ff:ff:ff:ff:ff link-netnsid 1
13: 14lxd3-0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether c6:67:ae:93:a0:68 brd ff:ff:ff:ff:ff:ff link-netnsid 0
15: 14lxd4-0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether d2:a8:30:fd:af:75 brd ff:ff:ff:ff:ff:ff link-netnsid 2
17: 14lxd5-0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether a2:b4:bc:f7:8d:da brd ff:ff:ff:ff:ff:ff link-netnsid 3
19: 14lxd0-0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether e2:95:a2:3c:2a:56 brd ff:ff:ff:ff:ff:ff link-netnsid 5
21: 14lxd2-0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP group default qlen 1000
link/ether ce:bb:56:c4:a7:b0 brd ff:ff:ff:ff:ff:ff link-netnsid 4
23: br-ex: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether c2:f7:c4:5d:3c:46 brd ff:ff:ff:ff:ff:ff
24: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 2e:08:8c:b8:aa:49 brd ff:ff:ff:ff:ff:ff
Connection to 192.168.7.41 closed.
Hi Drew,
Running the debug command on the nodes indicates issue with interface en02, confirming what I found with ovs-vsctl show.
unit-ovn-chassis-2: 23:36:22 WARNING unit.ovn-chassis/2.ovsdb-relation-changed subprocess.CalledProcessError: Command ‘[‘ip’, ‘link’, ‘set’, ‘eno2’, ‘up’]’ returned non-zero exit status 1.
unit-ovn-chassis-2: 23:36:22 ERROR juju.worker.uniter.operation hook “ovsdb-relation-changed” (via explicit, bespoke hook script) failed: exit status 1
I need to focus on fixing the interface problem and I will provide update.
Thanks.
Additionally, IP link on the nova-compute nodes shows the following:
admin@north:~$ juju ssh nova-compute/4 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens20f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-ens20f0 state UP mode DEFAULT group default qlen 1000
link/ether 54:ab:3a:70:d8:77 brd ff:ff:ff:ff:ff:ff
3: ens20f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 54:ab:3a:70:d8:78 brd ff:ff:ff:ff:ff:ff
4: lxdbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 00:16:3e:ac:6c:57 brd ff:ff:ff:ff:ff:ff
7: br-ens20f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 42:55:c9:f4:a1:52 brd ff:ff:ff:ff:ff:ff
8: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 9e:36:3e:5e:72:19 brd ff:ff:ff:ff:ff:ff
11: 12lxd2-0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP mode DEFAULT group default qlen 1000
link/ether 7a:47:12:2e:89:6a brd ff:ff:ff:ff:ff:ff link-netnsid 0
13: 12lxd4-0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP mode DEFAULT group default qlen 1000
link/ether ca:e4:cc:58:74:de brd ff:ff:ff:ff:ff:ff link-netnsid 1
15: 12lxd5-0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP mode DEFAULT group default qlen 1000
link/ether 42:55:c9:f4:a1:52 brd ff:ff:ff:ff:ff:ff link-netnsid 2
17: 12lxd6-0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP mode DEFAULT group default qlen 1000
link/ether 92:60:79:f3:08:8f brd ff:ff:ff:ff:ff:ff link-netnsid 3
19: 12lxd3-0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP mode DEFAULT group default qlen 1000
link/ether 6e:0d:0a:ce:01:8a brd ff:ff:ff:ff:ff:ff link-netnsid 4
21: 12lxd1-0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP mode DEFAULT group default qlen 1000
link/ether 6a:d0:42:46:7c:bc brd ff:ff:ff:ff:ff:ff link-netnsid 5
23: 12lxd0-0@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ens20f0 state UP mode DEFAULT group default qlen 1000
link/ether da:51:c3:cc:d7:c3 brd ff:ff:ff:ff:ff:ff link-netnsid 6
25: br-ex: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 26:ba:37:66:8e:4c brd ff:ff:ff:ff:ff:ff
26: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether ca:0f:7f:10:4a:b6 brd ff:ff:ff:ff:ff:ff
Connection to 192.168.7.18 closed.