Hi all,
what is the purpose for primary-network and external-network in a vSphere environment? I’m deploying Kubernetes clusters (with CDK and CORE) and thought the primary-network was used to communicate nodes between them and external for accesing from outside, but I can see, when I deploy the bundle, that is not in that way.
Thanks in advanced.
I had the same thoughts around this. We only use one network so we had the luxury to not understand this yet =D
I’m not sure what properties the primary network needs to have, like internet connectivity or what else… Etc.
The intent when modeling primary vs external was very much that. (Internal communication vs access from outside of the cloud.) I’m guessing anything that diverges from that tends to be “well I can route to external from everywhere, so I’ll just always use that” (shortcuts that weren’t intended).
It would be good to know where you’re seeing divergence from internal communication vs external communication.
Logical way is to use primary network to communicate between nodes inside, i.e., Kubernetes deployment and external network to gain access to Internet/other and to publish services.
When I declare the two kind of networks and deploy a CDK via juju the behavior is different at any time the bundle is deployed: sometimes the public address is the external one and others the public address is the primary in juju status.
By now, I’ll keep only the primary-network to deploy Kubernetes deployments, but it should be revised…
I know this is a bit of thread-necromancy, but I see the same behavior here - vSphere 7.0U2 + Juju 2.9.18
I have two networks defined:
primary-network: nv-22-jujuinternal
external-network: nv-23-jujuexternal
nv-22 is 10.50.22.0/24
nv-23 is 10.50.23.0/24
Both networks are routable and my client browser can hit IP’s in either subnet. There are no firewalls between either of these subnets, the subnet that vCenter is running on, or the subnet that I’m attempting to access the service from.
Both primary-network: nv-22-jujuinternal and external-network: nv-23-jujuexternal are configured as such in the controller bootstrap.yaml and in my model-config yaml which I’m passing prior to running juju deploy in a model.
When deploying juju-gui, I can see from status that “Public address” is showing as an address which is on the primary-network (nv-22)
Unit Workload Agent Machine Public address Ports Message
juju-gui/2* unknown idle 2 10.50.22.30 80/tcp,443/tcp
If I go to that address in a browser on http:// or https:// then I eventually get an ERR_CONNECTION_TIMED_OUT response, however if I go to vCenter and grab the IP that was provided via DHCP on the external-network then the Juju GUI loads up.
I see the same behavior with landscape-scalable too
Unit Workload Agent Machine Public address Ports Message
haproxy/0* active idle 0 10.50.22.31 80/tcp,443/tcp Unit is ready
landscape-server/0* active idle 1 10.50.22.32
postgresql/0* active idle 2 10.50.22.33 5432/tcp Live master (10.19)
rabbitmq-server/0* active idle 3 10.50.22.34 5672/tcp Unit is ready
Machine State DNS Inst id Series AZ Message
0 started 10.50.22.31 juju-dee5d2-0 bionic poweredOn
1 started 10.50.22.32 juju-dee5d2-1 bionic poweredOn
2 started 10.50.22.33 juju-dee5d2-2 bionic poweredOn
3 started 10.50.22.34 juju-dee5d2-3 bionic poweredOn
haproxy is listening on 80/443 on the ‘external-network’ address and responds when attempting access
https://10.50.23.21/new-standalone-user
but juju status haproxy shows the public address as being the DHCP assigned address from the primary-network
App Version Status Scale Charm Store Channel Rev OS Message
haproxy active 1 haproxy charmstore stable 61 ubuntu Unit is ready
Unit Workload Agent Machine Public address Ports Message
haproxy/0* active idle 0 10.50.22.31 80/tcp,443/tcp Unit is ready
Machine State DNS Inst id Series AZ Message
0 started 10.50.22.31 juju-dee5d2-0 bionic poweredOn
According to this here document I should be able to run juju status haproxy and use the IP address that it provides, but this has so far been false and I can only access 80 and 443 by going to vCenter and getting the external-network assigned address.
My Juju experience can be counted in minutes on one hand at this point, and it’s very possible that I’m just doing something wrong or missing a step, but from what I understand so far this isn’t expected or desired behavior?
Any advice would be most appreciated 
Thx
stu
Not sure how much info is too much info, but for what it’s worth, SSH seems to know which address to use:
Machine State DNS Inst id Series AZ Message
0 started 10.50.22.14 juju-799d5a-0 bionic poweredOn
1 started 10.50.22.15 juju-799d5a-1 bionic poweredOn
2 started 10.50.22.16 juju-799d5a-2 bionic poweredOn
3 started 10.50.22.17 juju-799d5a-3 bionic poweredOn
># juju ssh 0
Welcome to Ubuntu 18.04.6 LTS (GNU/Linux 4.15.0-162-generic x86_64)
~omitted for brevity~
Memory usage: 20% IP address for ens192: 10.50.22.14
Swap usage: 0% IP address for ens224: 10.50.23.13
and netstat to confirm
># netstat | grep 10.50.22.14
># netstat | grep 10.50.23.13
tcp4 0 0 10.169.16.249.50710 10.50.23.13.ssh ESTABLISHEDJuju tries all addresses to connect to a unit when wrapping ssh, which explains why it works.
@afreiberger actually gave me an idea on this post about DNS IP’s and so I did a bit of testing.
I swapped around the vSphere switches that were assigned to Juju, so
primary-network: (old) nv-22-jujuinternal --> (now)nv-23-jujuexternal
external-network: (old) nv-23-jujuexternal --> (now)nv-22-jujuinternal
It does appear that on vSphere the Public address of a unit is numerically the lowest available.
As a reminder to save people scrolling up
nv-22-jujuinternal == 10.50.22.0/24
nv-23-jujuexternal == 10.50.23.0/24
># juju model-config | grep network | grep nv
external-network model nv-22-jujuinternal
primary-network model nv-23-jujuexternal
># juju status
Unit Workload Agent Machine Public address Ports Message
haproxy/0* active idle 0 10.50.22.13 80/tcp,443/tcp Unit is ready
landscape-server/0* active idle 1 10.50.22.14
postgresql/0* active idle 2 10.50.22.15 5432/tcp Live master (10.19)
rabbitmq-server/0* active idle 3 10.50.22.16 5672/tcp Unit is ready
But then I thought there may be opportunity for a little more testing, so I swapped around the subnet range associated with each VLAN
Now, back to the original mappings of
primary-network: nv-22-jujuinternal
external-network: nv-23-jujuexternal
nv-22-jujuinternal == 10.50.23.0/24
nv-23-jujuexternal == 10.50.22.0/24
In theory, we should see that even though nv-22-jujuinternal is the primary network and the initial issue was that the primary interface was presented as the “Public address”, if my suspicion of “lowest numerical” is true, then the address associated with the external-network configurable should be presented by units as the public address, which now would be 10.50.22.0/24
Let’s test!
Waiting for address
Attempting to connect to 10.50.23.10:22
Connected to 10.50.23.10
~snip~
api-endpoints: ['10.50.23.10:17070']
># juju deploy landscape-scalable
~snip~
># juju status
Unit Workload Agent Machine Public address Ports Message
haproxy/0* active idle 0 10.50.22.13 Unit is ready
landscape-server/0* active idle 1 10.50.22.14
postgresql/0* active idle 2 10.50.22.15 5432/tcp Live master (10.19)
rabbitmq-server/0* active idle 3 10.50.22.16 5672/tcp Unit is ready
And there we go, lowest numerical wins for the “Public address” on vSphere.
Now, on the downside, haproxy is accessible now on the haproxy decided it didn’t want to expose ports there, but I suspect that’s likely due to the charm not handling multiple interfaces?private-network IP…
Anyway, I think at this point I should probably file an issue report vs. filling up this thread. That’ll be the next thing I go learn how to do.
Thx
stu