I’m presently working on an issue where I need to manage multiple sets of LXD containers via Juju. Each set of containers would be associated with a different LXD network. We would be starting with 2 such sets, but want the flexibility to add more sets later if needed. I’m trying to find the best way of doing this.
Currently what I have in mind is: we could split the containers into separate models. Each model would have its network defined in a space, and each model would set “juju model-config default-space” appropriately, resulting in containers launched on the models being landed on those spaces.
Question 1: How do we appropriately set up controller connectivity in this situation? Do we land the controller directly on each of the target network spaces, and if additional spaces are required later, somehow add those spaces to the existing controller later? Or do we have a separate space just for controller-to-agent communication, and land the controller purely on there, and land all the non-controller containers on the controller space in addition to their target space?
Question 2: I’ve noted that if I have containers landed on 2 spaces (and thus 2 LXD bridges), I end up with default routes being set for both created adapters. Is there a way to only have one default route be kept, or does this require handling at the charm level?
Q1)
The requirement for connectivity to the juju controller is just that the space the controller is in is routable from the space that the workloads are in, not that they share a space.
Generally having a clear ‘this is where the controllers are’ space is useful, since it makes the rest of the logic clearer.
Q2) For default routes it does get a bit tricky, because how do you know which route should become the ‘more default’ one. I don’t think it is something that Juju itself is modeling yet.
Thank you John - that’s a very good point regarding routing.
I think we’ll do something like this:
In the per-model LXD profiles, we can override the network adapter so that it will use the LXD network bridge of our liking, and not use whatever might be defined in the “default” profile. This basically would provide us with different networks for each model as we desire.
By default, LXD bridges have routing enabled and can route between each other. So, we can land our LXDs on single networks, avoiding issues with multiple default routes, and rely on routing between the LXD bridges to enable communication with the controller.
We can use firewall rules to restrict this routing as appropriate.
I’m open to other ideas, but I think this may be sufficient. Thanks.
In the event that your controller is connected to multiple spaces and you want to ensure that a particular one is used by agents, you can also configure the juju-mgmt-space, which is the space that is used to filter the API addresses that get set in agent config.
So assuming the spaces are routable between each other, you can be be prescriptive about traffic to that extent.
I know this is an old thread, but it seems it’s still happening on the latest stable of Juju.
Right now, juju seems to use a really old syntax of netplan, where gateway4 is still used to define the default route. Netplan has moved away from that for some time now, in favor of the routes option.
It’s completely okay to have multiple default gateways. MAAS models this quite well, by setting one of the gateways in the default routing table (254) and the rest of them in different route tables (1, 2, etc).
The above sets the default route in the default routing table via 192.168.100.1. The second gateway which is 192.168.122.1 is set as a default route, but in table 2. And a rule for the host route to and from 192.168.122.0/24 is also added to the default routing table.
The result of this is that you avoid asymmetric routing. Traffic you get through the 192.168.122.1 gateway, will be sent back out through the same gateway. Traffic you get from peers on the same L2 network will be sent out through the same interface.
This config also allows the agent to communicate with the controller, whereas having multiple gateway4 statements, all in the default routing table can lead to asymmetric routing which will flap the connection of the agent to the controller, and you’ll see it do down and up periodically.
