Openstack bootstrap mongodb issue (spaces?)

juju
1

This seems complex enough to need some documentation, which I can’t find. :slight_smile:

I did manage to bootstrap a controller in Openstack but I came across this…

routergod@juju:~$ juju bootstrap killstack --metadata-source=/home/routergod/simplestreams/images --model-default network=7d038caf-0679-4c12-8595-71a63e8d3def --model-default external-network=4e0cbeb9-917d-4444-a75d-3452522c1d34 --model-default use-floating-ip=true --debug --config bootstrap-timeout=3600

Lots of output, and then;

2020-09-15 18:20:54 DEBUG juju.mongo open.go:160 mongodb connection failed, will retry: dial tcp 127.0.0.1:37017: connect: connection refused
2020-09-15 18:20:55 DEBUG juju.mongo open.go:174 dialled mongodb server at "127.0.0.1:37017"
2020-09-15 18:20:55 INFO juju.replicaset replicaset.go:58 Initiating replicaset with config: {
  Name: juju,
  Version: 1,
  Protocol Version: 0,
  Members: {
    {1 "172.16.2.168:37017" juju-machine-id:0 voting},
  },
}
2020-09-15 18:21:00 INFO juju.replicaset replicaset.go:60 Unsuccessful attempt to initiate replicaset: No host described in new configuration 1 for replica set juju maps to this node

It blows up here after a timeout. Mongodb apparently can’t talk to itself? Snooping on the network (on neutron-gateway unit) I can see this (note 172.16.0.0/16 is my external network in this case);

18:22:27.349956 IP 198.51.100.132.46628 > 172.16.2.168.37017: Flags [S], seq 4072893471, win 65228, options [mss 1418,sackOK,TS val 1073724551 ecr 0,nop,wscale 7], length 0
18:22:27.350017 IP 172.16.2.168.46628 > 198.51.100.132.37017: Flags [S], seq 4072893471, win 65228, options [mss 1418,sackOK,TS val 1073724551 ecr 0,nop,wscale 7], length 0
18:22:29.365415 IP 198.51.100.132.46628 > 172.16.2.168.37017: Flags [S], seq 4072893471, win 65228, options [mss 1418,sackOK,TS val 1073726566 ecr 0,nop,wscale 7], length 0
18:22:29.365546 IP 172.16.2.168.46628 > 198.51.100.132.37017: Flags [S], seq 4072893471, win 65228, options [mss 1418,sackOK,TS val 1073726566 ecr 0,nop,wscale 7], length 0

Neutron is doing some source-NAT thing here which seems to put the flows outside the contract defined in the security group that bound to the controller instance.

I can bodge this by adding a rule to the security group to permit traffic from 172.16.2.168 (i.e. the controller’s float IP). If I do this while the bootstrap is running, it completes ok.

I expect I should be doing something with juju spaces to resolve this but I could sure use a clue.

2

What kind of networking setup do you have? Specifically curious about the external network in this configuration. Typically, if you’re using floating IP’s (as indicated with the use-floating-ip=true in the bootstrap command) then I’d expect that the instance hosting the Juju controller would be launched on your private network and wouldn’t have visibility into the external network as the floating IP is NAT’d. However, seeing the Juju configuring the mongo replicaset your external network makes me curious about the external network here.

3

This is a disposable test model, the addressing is a bit weird. 172.16.0.0/16 is my Openstack external network (it is actually a LAN connected to a broadband router). I have allocated 172.16.2.0/24 for floats and whatnot in this model environment.

Inside Openstack I just have the default admin project, with a network defined by my scripts (198.51.100.0/24, a bogon) where the Juju controller is deployed. A gateway router connects this network to the external network.

routergod@juju:~$ openstack router show -fyaml provider-router
admin_state_up: UP
availability_zone_hints: ''
availability_zones: nova
created_at: '2020-09-10T20:10:18Z'
description: ''
distributed: false
external_gateway_info: '{"network_id": "4e0cbeb9-917d-4444-a75d-3452522c1d34", "enable_snat":
  true, "external_fixed_ips": [{"subnet_id": "c4471b9b-26ae-44c3-b5ef-434e1d7b2d95",
  "ip_address": "172.16.2.226"}]}'
flavor_id: null
ha: false
id: 673fb338-4301-4f35-b9e3-3e07f31440f4
interfaces_info: '[{"subnet_id": "15133f47-3af8-446b-8952-a28646ddae39", "ip_address":
  "198.51.100.1", "port_id": "d838264c-e424-44d3-8ad9-51af911e68e6"}]'
name: provider-router
project_id: 5a374f98fa9141b582940bc3aa7e28fb
revision_number: 4
routes: ''
status: ACTIVE
tags: ''
updated_at: '2020-09-10T20:10:33Z'

My Juju client machine is connected directly to the ‘external’ network 172.16.0.0/16. I expose the Openstack API via an nginx connected to the same external network.

4

If you run this same bootstrap with --keep-broken, I’d be interested to see what the instance addresses are.

The logic retrieves the the instance addresses and should select a cloud-local address before a public one for initialising the replica-set, which would tend to indicate that the floating IP was the only one returned or that all the returned addresses were public.

5

Thanks for your interest.

routergod@juju:~$ openstack server show -fvalue -caddresses adf547ec-9865-47d3-b66b-e261b946f9d3
internal=198.51.100.76, 172.16.2.168
6

Would defining spaces for your juju model for these two networks as internal-space and public-space help to be able to distinguish at the charm level which interface is for use by which binding?

Then you might find deploying mongodb with the binding “replica-set=internal-space” would establish the intra-cluster communication on the private network space, and the public network could then be bound to the mongodb or other external charm interface(s) you want to relate to/serve data on.

I’m wondering if the spaces are being defined for you by nature of the bootstrap options “network” and “external-network”, but not used in your deployment bundle.yaml explicitly to separate the traffic as intended.

After bootstrap, what is the output of “juju spaces”?

7

Yes that sounds plausible I will give it a go. By default juju has everything in alpha.

Name   Space ID  Subnets
alpha  0         172.16.0.0/16
                 198.51.100.0/24
                 252.0.0.0/8
                 253.0.0.0/8
8

@afreiberger I really thank you for your suggestion here but I am going around in circles. How do I define spaces at controller bootstrap?

9

I’m honestly not certain spaces can be defined for openstack provider during bootstrap, however, there is good reference on managing spaces here under " Modifying spaces": Juju | Network spaces

Specifically:

$ juju move-to-space db-space 172.31.16.0/20
Subnet 172.31.16.0/20 moved from alpha to db-space
10

@afreiberger Many thanks. It appears I have been barking up the wrong tree here. I just tried the same bootstrap using an Openstack internal network with an RFC1918 range rather than the 198.51.100.0/24 bogon. Same RFC1918 external network. This works fine.

@manadart the logic you refer to sounds ‘interesting’ :slight_smile: