For the upcoming 2.9 version of juju, we have introduced a few changes that will not only provide charm authors with the ability to open port ranges to specific application endpoints but also give give operators the power to control which application endpoints get exposed and who (spaces or CIDRs) should be able to access the ports opened for those endpoints.
The purpose of this post is to introduce this new feature from the perspectives of charm authors and juju cloud operators and provide a few examples of what can be achieved using it.
As always, if you are interested in test-driving the new feature, you can switch to the latest edge snap by running snap refresh --channel latest/edge juju.
If you do decide to try it out, please let us know what you think of this new feature!
Opening ports for specific endpoints
In previous versions of juju, when a charm requests a port to be opened, the port is implicitly opened for all defined application endpoints.
Opening and closing ports or port ranges is typically handled by the framework used by charm authors. Under the hood, such requests trigger an invocation of the open-port, opened-ports and close-port hook tools.
Starting with juju 2.9, the aforementioned hook tools support an optional --endpoints flag which allows the charm to constrain the opened port range to a comma-delimited list of application endpoints. If the flag is not provided, the command behaves in exactly the same manner as previous versions of juju, i.e. it opens the port for all application endpoints.
Opening ports
In the following example, we deploy percona-cluster and then proceed to open port 3306 for all endpoints and port 8080 for the db-admin endpoint.
$ juju deploy percona-cluster
$ juju run --unit percona-cluster/0 "open-port 3306/tcp"
$ juju run --unit percona-cluster/0 "open-port --endpoints db-admin 8080/tcp"
The opened-ports tool, when invoked without any argument, will by default display the unique set of opened ports (for all endpoints) opened by the charm. This is intentional as we don’t want to break any existing charms out there that may be parsing the output of this command and assuming that the output is always formatted in a particular way.
$ juju run --unit percona-cluster/0 "opened-ports"
3306/tcp
8080/tcp
If we pass the --endpoints flag to the above command, we will get back the extended opened port range report which is broken down by endpoint. In the following output the * symbol represents all endpoints.
$ juju run --unit percona-cluster/0 "opened-ports --endpoints"
3306/tcp (*)
8080/tcp (db-admin)
The semantic meaning of “all endpoints”
If the charm needs to open a particular port range for all endpoints, it can do so in two ways:
- Via an
open-port port/protocolinvocation. - Via an
open-port --endpoints endpoint-list port/protocolinvocation where theendpoint-listvalue is a comma-delimited list of all application endpoints.
While both approaches are functionally equivalent (i.e. the port range gets opened for all endpoints), there is a slight difference with respect to the charm author’s stated intent.
The first command requests juju to open the port range for all endpoints, present and future. In other words, if the charm gets upgraded and the new version defines new endpoints, the port range will be automatically opened for the new endpoints as well.
On the other hand, the second command provides an explicit list of endpoints for which the port range should be opened. If an newer version of the charm defines new endpoints, the port range will not be automatically opened for them.
Closing ports
Analogous to the open-port tool, close-port also allows an endpoint list to be specified. As you might expect, if the --endpoints flag is omitted, the specified port range will be closed for all endpoints.
Continuing with the percona-cluster example from above, let’s now close port 3306 and list the open port ranges again; then close port 8080 (note that no --endpoints flag is specified) and list the open port ranges one more time:
$ juju run --unit percona-cluster/0 "close-port 3306/tcp"
$ juju run --unit percona-cluster/0 "opened-ports --endpoints"
8080/tcp (db-admin)
$ juju run --unit percona-cluster/0 "close-port 8080/tcp"
# No output will be returned as no ports are opened
$ juju run --unit percona-cluster/0 "opened-ports --endpoints"
What if the charm needs to open a port range for all endpoints except one? The open/close port mechanism also supports this particular use-case:
$ juju run --unit percona-cluster/0 "open-port 3306/tcp"
$ juju run --unit percona-cluster/0 "opened-ports --endpoints"
3306/tcp (*)
$ juju run --unit percona-cluster/0 "close-port --endpoints db-admin 3306/tcp"
$ juju run --unit percona-cluster/0 "opened-ports --endpoints"
3306/tcp (access, cluster, db, ha, master, nrpe-external-master, shared-db, slave)
As you can see, the close-port invocation caused juju to replace the “all endpoints” entry with the list of all application endpoints except the db-admin one.
Supporting open/close ports for endpoints when writing new charms
Charm authors that are interested in leveraging the new --endpoints flag when invoking the aforementioned hook tools must first check whether the flag is supported by the environment where the charm is deployed to by inspecting the value of the JUJU_VERSION env-var and deciding based on it value (2.9+) whether this feature is available to them.
Granular control of application expose setting by operators
In the second part of this post we will explore the improved expose feature from the perspective of the cloud operator.
When an application was exposed in a previous version of juju, either via the CLI (e.g. juju expose APP) or via a bundle with an exposed: true field, all of its opened port ranges were automatically made accessible from everyone (i.e. the ingress rules used 0.0.0.0/0 as the source CIDR for incoming traffic).
From juju 2.9 and onwards, operators can opt to either expose all application ports or to only expose the ports that charms have opened for a particular list of application endpoints. Moreover, the operator can specify both globally, and on a per-endpoint basis, a set of source CIDRs (and/or spaces) that should be allowed to access the opened port ranges that have been opened either globally or for the specified endpoint.
Controlling expose settings via the juju CLI
Exposing individual application endpoints
As of juju 2.9, the juju expose command now accepts the following optional flags:
-
--endpoints: a comma-delimited list of endpoints to use for selecting the list of port ranges to be exposed. -
--to-cidrs: a comma-delimited list of CIDRs that should be able to access the selected port ranges. -
--to-spaces: a comma-delimited list of space names that should be able to access the selected port ranges.
Note that all of the above flags are optional. If none of these flags are provided (e.g. juju expose percona-cluster), juju will instead evaluate the following equivalent command juju expose percona-cluster --to-cidrs 0.0.0.0/0,::/0 which makes all opened ports accessible from everyone thus matching the behavior of older juju versions.
On the other hand, if a list of endpoints is specified but no other flags are provided (e.g. juju expose percona-cluster --endpoints db), juju will once again assume an implicit --to-cidrs 0.0.0.0/0,::/0 argument.
Contrary to older juju versions where juju expose X is an one-off thing that simply marks the application as exposed, with juju 2.9, you can execute a sequence of juju expose X commands to specify expose settings for individual application endpoints. Note that each juju expose X --endpoints command will overwrite the previous expose settings for each referenced endpoint name.
Let’s take a look at a simple example where we expose all endpoints of percona-cluster and then override the expose settings for the db-admin endpoint to only allow access from the local network (10.0.0.0/24). Finally, we repeat the last expose command to specify a different set of CIDRs.
$ juju expose percona-cluster
$ juju expose percona-cluster --endpoints db-admin --to-cidrs 10.0.0.0/24
$ juju show-application percona-cluster
percona-cluster:
...
exposed: true
exposed-endpoints:
"":
expose-to-cidrs:
- 0.0.0.0/0
- ::/0
db-admin:
expose-to-cidrs:
- 10.0.0.0/24
...
$ juju expose percona-cluster --endpoints db-admin --to-cidrs 192.168.0.0/24,192.168.1.0/24
$ juju show-application percona-cluster
percona-cluster:
...
exposed: true
exposed-endpoints:
"":
expose-to-cidrs:
- 0.0.0.0/0
- ::/0
db-admin:
expose-to-cidrs:
- 192.168.0.0/24
- 192.168.1.0/24
...
Un-exposing individual endpoints or the entire application
In a similar fashion, the juju unexpose command also supports an optional --endpoints flag which may be specified to completely remove the expose settings for a list of endpoints. Juju will automatically mark the application as unexposed when the last expose setting is removed.
If the command is invoked without the --endpoints flag, all expose settings will be deleted and the application will be marked as unexposed.
Providing expose settings via bundles
Specifying all the expose rules for each application in your model can be quite tedious. Fortunately, we can use a bundle to apply all rules in a single go! As the per-endpoint expose settings are deployment-specific, they must always be specified as part of an overlay.
Furthermore, when your bundle contains an overlay section with expose settings, the expose field (with a true value) is not allowed to be present anywhere inside the bundle as this can cause security issues with older controllers. To understand why this is the case, let’s examine the following bundle and its overlay:
series: bionic
applications:
percona-cluster:
charm: cs:percona-cluster-291
num_units: 1
to:
- "0"
# This not allowed and is only included for this example!
expose: true
machines:
"0": {}
--- # overlay.yaml
applications:
percona-cluster:
exposed-endpoints:
"":
expose-to-cidrs:
- 0.0.0.0/0
- ::/0
db-admin:
expose-to-cidrs:
- 192.168.0.0/24
As you can see above, all endpoints of the percona-cluster applications are exposed to the world with the exception of ports opened for db-admin which are only accessible by 192.168.0.0/24.
If we were to take this bundle, as-is, and deploy it to a 2.8 controller, the controller would ignore the exposed-endpoints section of the overlay and only consider the expose: true entry. This would make all ports (including the db-admin ones) accessible from 0.0.0.0/0 which is clearly not the
operator’s intent!
To prevent this from happening, juju will refuse to deploy the bundle if it contains both an expose: true flag and an exposed-endpoints section and will instead display an error:
ERROR cannot deploy bundle: the provided bundle has the following errors:
exposed-endpoints cannot be specified together with "exposed:true" in application "percona-cluster" as this poses a security risk when deploying bundles to older controllers
Furthermore, when exporting bundles via juju export-bundle, the controller will populate the exposed-endpoints section for applications that include endpoint-specific expose settings and omit setting the expose flag. On the other hand, if an application exposes all endpoints to 0.0.0.0/0, the bundle exporter will instead set the expose flag and omit the exposed-endpoints section.
This behavior follows the principle of least surprise. If the bundle is exported from a 2.9 controller and subsequently deployed to a 2.8 controller, the application will not be exposed (as there is no expose: true field present) out of the box to the entire world. Instead, the operator will need to step in, evaluate the risk involved and manually expose the application if needed.
Technical details: How ingress rules are generated
Instead of keeping track of a single boolean flag that indicates whether an application is exposed or not, juju 2.9 maintains, for each exposed application, a list of expose settings for each individual application endpoint. This list also contains a special entry that applies to all endpoints.
Each entry of the list includes the following bits of information:
- A list of CIDRs that should be able to access any port ranges opened for a particular endpoint.
- A list of spaces that should be able to access the same set of ports.
Juju consults this information whenever it needs to regenerate the ingress rules for a deployed unit. For each endpoint in the expose settings list, juju applies the following steps:
- Calculate the list of port ranges to be exposed. For individual endpoints, this is the union of ports explicitly opened by the charm for the endpoint and any ports that the charm has opened for all endpoints. For the special “all endpoints” entry, this is the union of any ports that the charm has opened for all endpoints and the ports opened for any individual endpoints that do not have endpoint-specific entries in the list.
- Calculate the list of source CIDRs by taking the union of any operator-specified CIDRs and the CIDRs that correspond to the subnets of each space that should be able to access the selected ports.
Using these two sets, juju will generate an ingress rule for each port range and CIDR combination.
Juju automatically regenerates the ingress rules for individual units when:
- The application expose settings change.
- The charm opens/close port ranges.
- The space topology changes (spaces get removed, subnets added/removed or moved to different spaces). This only affects applications that contain expose settings that include spaces.