I’m presently working on an issue where I need to manage multiple sets of LXD containers via Juju. Each set of containers would be associated with a different LXD network. We would be starting with 2 such sets, but want the flexibility to add more sets later if needed. I’m trying to find the best way of doing this.
Currently what I have in mind is: we could split the containers into separate models. Each model would have its network defined in a space, and each model would set “juju model-config default-space” appropriately, resulting in containers launched on the models being landed on those spaces.
Question 1: How do we appropriately set up controller connectivity in this situation? Do we land the controller directly on each of the target network spaces, and if additional spaces are required later, somehow add those spaces to the existing controller later? Or do we have a separate space just for controller-to-agent communication, and land the controller purely on there, and land all the non-controller containers on the controller space in addition to their target space?
Question 2: I’ve noted that if I have containers landed on 2 spaces (and thus 2 LXD bridges), I end up with default routes being set for both created adapters. Is there a way to only have one default route be kept, or does this require handling at the charm level?
Q1)
The requirement for connectivity to the juju controller is just that the space the controller is in is routable from the space that the workloads are in, not that they share a space.
Generally having a clear ‘this is where the controllers are’ space is useful, since it makes the rest of the logic clearer.
Q2) For default routes it does get a bit tricky, because how do you know which route should become the ‘more default’ one. I don’t think it is something that Juju itself is modeling yet.
Thank you John - that’s a very good point regarding routing.
I think we’ll do something like this:
In the per-model LXD profiles, we can override the network adapter so that it will use the LXD network bridge of our liking, and not use whatever might be defined in the “default” profile. This basically would provide us with different networks for each model as we desire.
By default, LXD bridges have routing enabled and can route between each other. So, we can land our LXDs on single networks, avoiding issues with multiple default routes, and rely on routing between the LXD bridges to enable communication with the controller.
We can use firewall rules to restrict this routing as appropriate.
I’m open to other ideas, but I think this may be sufficient. Thanks.
In the event that your controller is connected to multiple spaces and you want to ensure that a particular one is used by agents, you can also configure the juju-mgmt-space, which is the space that is used to filter the API addresses that get set in agent config.
So assuming the spaces are routable between each other, you can be be prescriptive about traffic to that extent.
