How to reconcile config.yaml's deliberate restrictiveness with hierarchical app config

framework
1

I had aked a question in this Charming Docs post but later realized that I should’ve asked the question here. So I’m re-asking here.

How does a charm author reconcile config.yaml's deliberate restrictiveness with an application’s hierarchical configuration? For example, AlertManager has a non-flat configuration file (example) and translating some of the options to config.yaml would be non-trivial.

For the simpler parts, it would obviously just be a matter of flattening the options in the config.yaml file. For example, the following AlertManager configuration options:

global:
  smtp_smarthost: 'localhost:25'
  smtp_from: 'alertmanager@example.org'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'password'

Would be defined in the charm’s config.yaml as

options:
  global_smtp_smarthost:
    type: string
  global_smtp_from:
    type: string
  global_smtp_auth_username:
    type: string
  global_stmp_auth_password:
    type: string

But how would one deal with repeating options such as AlertManager’s receivers? For example, I tried to map the following to what was possible with config.yaml

receivers:
- name: 'team-X-mails'
  email_configs:
  - to: 'team-X+alerts@example.org'

- name: 'team-X-pager'
  email_configs:
  - to: 'team-X+alerts-critical@example.org'
  pagerduty_configs:
  - service_key: <team-X-key>

...

The “best” approach I could get with config.yaml was

options:
  receivers1_name:
    type: string
  receivers1_config_type:
    type: string
  receivers1_config_options:
    type: string  # Would really have to be a minified JSON string or a base64 encoded YAML string
  receivers2_name:
    type: string
  receivers2_config_type:
    type: string
  receivers2_config_options:
    type: string  # Would really have to be a minified JSON string or a base64 encoded YAML string
...

Obviously the above approach is not ideal for a number of reasons:

  1. The number of receivers defined would be limited to however many receiversX_* is defined in config.yaml
  2. Dropping down to a JSON string or a base64-encoded YAML string defeats the purpose of config.yaml's restrictiveness which is to simplify AlertManager configuration
  3. As per the documentation referred to above: “If you’re considering using base64 encoding to slip structured data through the deliberately restrictive configuration language, you’re probably ‘Doing It Wrong.’”
  4. Note in the AlertManager config above how the team-X-pager receiver has a config for both email and pagerduty. How does one represent that in config.yaml?

My initial analysis to the above problem is that it’s a symptom of me trying to create an “Unnecessary Abstraction Over An Abstraction™.” AlertManager’s configuration is already an abstraction over a complex problem domain and it is, for now, the simplest way to configure alerting. To try and abstract that with options in config.yaml seems unnecessary and a side effect of this would be that the charm user would have to learn the configuration options of the charm and try and map that to AlertManager’s own configuration options. This means a lot of jumping back and forth between the charm’s documentation and AlertManager’s documentation. Again, that defeats the intended purpose of conig.yaml and charming in general which is to simplify the job of deploying and operating AlertManager or any other application.

Of course, I haven’t stopped my analysis there. To move forward with my charming work, I tried to prototype a number of approaches. One of the approaches that I found promising was making use of juju’s --resource option. This GitHub Pull Request already explains this thought process well enough.

I am also wary of the possibility that I might just be missing some hidden powers behind config.yaml's parser so I’d be more than happy to be corrected of my (mis)understanding of the above restrictiveness.

I would love to know what the rest of the community thinks about this. Thank you so much for reading this far.

4 Likes
2

I’ve seen some charms include a free-form or semi-structured config directive like cinder’s config-flags option.

And I’ve seen some charms use resources, such as cinder’s policy-override.

In this case they serve different purposes but have similar goals: get somewhat arbitrary text into a config file.

I don’t know which is better from a programmatic perspective but from a deployer’s perspective I can see advantages from either approach. The config-flags style is nice for small snippets because it can all be contained within your bundle. The resource approach is nice because the file is maintained separately from the bundle and you don’t have to worry about the yaml formatting of the bundle. This is more practical for larger inclusions.

If you could change how config.yaml functions, is there a way to implement something like the multiple receiver options for your AlertManager charm? Perhaps a way to specify a wildcard that would roll up into an array within the charm code? Something like:

  options:
    receivers<num>_name:
      type: string
    receivers<num>_config<num>_type:
      type: string

Then, in someone’s config, they could add numbers to have an arbitrary number of receivers and receivers configs. Or they could leave the numbers off and have just one receiver/config.

Seems like this would be a significant update to how config.yaml is parsed so maybe this is unrealistic.

3

TBH, the programmatic perspective should take a backseat here in favor of what makes sense from the operator/deployer’s PoV. In fact, I’d totally avoid thinking about the programmatic perspective until the user experience is properly defined.

Agreed. To add, the biggest advantage I see with this is that those experienced with configuring the underlying application don’t have to unnecessarily re-learn new configuration options that would just get re-mapped by the charm to the config schema that the experienced user already knows.

Indeed, that would be a big change. I don’t know if it’s worth the effort or not but at least I know that something needs to change either in Juju or whatever charming framework one is using such that it addresses this case because learning an unnecessary new config option can be a blocker to charm/operator adoption.

3 Likes
4

For reference, in OperatorHub.io, the Prometheus/AlertManager operator exposes AlertManager’s configuration directly and does not try to abstract it out. See their documentation. This seems to be the same for other operators like that of Grafana.

5

@mmaglana I think you’ve touched on a valid concern with Juju’s current config solution. I think that the simple config that Juju currently supports is probably most often insufficient for full blown configuration of an application. Unless an applications config is made of simple key-value pairs, I think it is probably most reasonable to just write a native application config file and then include it as either a base64 encoded string or a Juju resource. I agree that it doesn’t make sense to try and make the app’s config “fit” into the Juju config.yaml.

Between base64 encoding and file resource I kind of feel like base64 encoding the file provides the easiest method of getting it in, but it isn’t exactly the perfect user-experience. Resources are fine, but like was brought up in your PR, it does limit your options from a bundle perspective.

Oh, I just had an idea. What if it was built into the juju config system where you could mark a config item as being a “file” type instead of a “string” type in your config.yaml. Then when you use juju config alertmanager config-file=base64 alertmanager-config.yaml it would know that the config-file config setting should reference a file and it will automatically base64 encode the file and include it int the config.

The config-get hook tool would then automatically decode any “file” type configs into strings, maybe.

6

Thank you so much for your response @zicklag!

Your suggestions would be great to have on juju! Having the base64 encoding transparent to the charm would be useful as well. However, I think that the path of least surprise would be to present to the charm a path to the file that has already be base64 decoded since it would be strange for an config option to have a type of file but is presented as a string to the charm. I may be wrong though so I’m open to correction. Either way, having a more flexible config.yaml schema that accommodates this non-edge-case scenario would be a great move!

2 Likes
7

Ah, I hadn’t thought of that. That makes a lot of sense actually. It’s very intuitive and properly handles binary files ( not that I’ve ever heard of a binary config file, but you never know ). It makes it sort of a hybrid of a config and a resource, but I think it very nicely handles the issue of application config files.

Actually, though, is it too much like a resource? Should we just allow bundling resources as base64 inside of a bundle YAML? I’m wondering if that actually make a bit more sense. Because we are essentially emulating resources inside of configs at that point.

8

If resources could be bundled just like config options, I think that would be an even better solution.

2 Likes
9

There might have to be a size limit on the bundled resources because a Juju resource could be used to distribute a multi-gigabyte file, but it would not make sense to have a multi-gigabyte YAML file.

10

Oh, I didn’t realize that this was in the Juju documentation.

I think that this is misleading and, frankly, incorrect, given that there is not a much better alternative to using base64 encoding if you want to be able to export a bundle.yaml with structured config in it.

I think that should probably be changed. Simple config is fine, but if an application needs complicated configuration, piping it through the overly restrictive simple config is not going to make it any simpler.

2 Likes
11

One example of a charm with configuration too complex to be represented in a structured way config.yaml, but where a straight base64-encoded config doesn’t quite work either, is cs:haproxy.

The charm makes heavy, invaluable use of relations, but it was clearly designed to infer most of its configuration from those relations: relating a website creates both frontend and backend stanzas in haproxy.cfg, with server details and options coming from the remote end of the relation. But the backend services are the least interesting part of configuring haproxy; having primarily relation-oriented configuration is backwards. This means you end up with services’ general http interfaces emitting haproxy-specific config options, and custom request routing (which is necessary in any non-trivial haproxy website or webservice deployment) is awkward and often requires modifying backend services to shovel the right deployment-specific data over the relation, when all that really should be coming from the other side is a set of ip:port pairs (plus maybe a concurrent request limit, or other unit-specific things like that).

Over the years the haproxy charm has grown the services option, which allows new stanzas to be defined and options overridden with almost literal haproxy config syntax. But it continues to try to simplify the configuration language: each service entry still creates frontend and backend stanzas, and there’s a hardcoded map of which option goes into which stanza, neither of which is what you always want and which require gross hacks to do what you actually need. In the Snap Store deployment we did a similar thing to cs:squid-reverseproxy except took it one step further to allow disabling implicit generation of stanzas from relations and even rename passed-through relations, for when we had a big multi-service haproxy but only wanted squid to reexport a subset of its services.

Charms often try to be opinionated and avoid exposing users to the complication of the underlying software. But the software didn’t grow that complication for no reason, so it’s frustrating that when I step outside the bounds of what the charm author expected I have to add a new config option, which inevitably just writes the config option out verbatim to the relevant bit of the configuration file. We’ve also seen this in charms that don’t make as much use of relations, e.g. cs:postgresql where dozens of deployment-specific tuning options like default_statistics_target are deprecated in favour of the literal extra_pg_conf option, and the charm’s both cheaper to maintain and more useful for that change.

I think what I really want from the haproxy charm is a templated literal config. A relation shouldn’t magically expose a port, but it should show up in the template variables so I can easily generate a server line for each remote unit. I’ve wondered in the past whether user-configurable options on relations might help, but they don’t solve e.g. the problem in haproxy of having a single frontend routing to a bunch of backends (there’s no Juju object that has a one-to-one mapping with frontend stanzas). So I think exposing the full power of the underlying configuration language is the only sensible option.

This obviously prevents a simple juju deploy wordpress, juju deploy haproxy, juju add-relation wordpress haproxy from doing anything useful, but the community seems to be leaning more and more towards bundles for that sort of thing anyway. The bundle would then just have to include a small haproxy config template to expose the units from the relation on some port. This is another case that shows how charms probably aren’t the right place for high-level logic or policy decisions, and they belong at another level like a bundle or meta-operator.

4 Likes
12

This already mostly exists if you’re using bundles. We always set non-trivial values of haproxy’s services option using include-file://, and make heavy use of include-base64:// elsewhere. It’s a bit unfortunate that the charm has to decode the latter itself, though.

We’ve also thought about the parallels between config and resources. There are really a couple of different types of resources: those that are part of the charm or a closely tied payload that’s provided by the Charm Store and never overridden, and those that are operator configuration that might or might not have a sensible default. From a quick look, Spark is a charm that seems to have both: its bigtop-repo resource is part of the payload, while sample-data is clearly a default dataset that would usually be overridden. It doesn’t seem ideal that those two types of files are conflated, and both separate from the YAML config which can have options in the hundreds of lines or that themselves include files.

2 Likes
13

Thanks for the additional context, @wgrant. I want to highlight the most important part (at least for me) in your response:

14

I’ve come to the conclusion that having charms named after a multi-purpose tool (like apache, haproxy, nginx, or squid) is problematic. Each of these tools can be configured to fulfill different roles (e.g. file server, load balancer (API or streaming), forward/reverse proxy with or without caching, rate-limiting/shaping, etc). Trying to engineer one charm that can have all appropriate tuning knobs exposed in a meaningful way is an exercise in frustration and poor UX.

If instead, you had charms that were based on their roles (e.g. API load balancer), which maybe used haproxy under the hood, then the juju config would have much smaller set of opinions to have, and is more tractable in general. It would also use more specific interfaces that just http, again providing better semantics to the consuming charms.

Regards haproxy in particular, I agree with @wgrant’s criticism (we’re on the same team). As an example of the above idea, I’ve long wanted to write an API load balancer charm, based on HAProxy, and using a proper parser, along with new interface type, to be able to better express our desired configuration. Part of this would likely include template literals in the juju config, albeit not entire files, as there’s still a need to amalgamate information from juju relations.

15

FWIW, we have talked quite a bit about supporting structured config that would allow for lists of objects. It was part of a wider discussion that would also give space for charms to have ‘output variables’ that would be part of that structure.
The last active discussion was from a while ago, with the discussion that we wanted to drive configuration of snaps and learn what works well when it isn’t a distributed case, and then take the lessons learned there to drive it into the distributed case.

There has also been discussions about representing ‘services’ as a concept (it was some of what motivated the rename from services to application in the Juju 2.0 release). A service would in some ways be a subset of a given application, but also would be cross-cutting potentially between applications. (this is the subset of these applications that are responsible for providing this logical service to this end-user, which is this part of the database, this part of the application tier, this part of the ha proxy front end.) each Service would then come with a bit of config for each application and some structured output for that service.

This is all still theoretical but it was an interesting way to model it.

16

In response to the last 2 comments, I have the feeling that indeed stopping to talk about charms with a name that they represent, but start talking about a more abstract service could work really well.

From my experience with modelling things using Archimate, it really works well to only talk about services in the “Application Layer”. This does feel a bit force when starting, but it gives the flexibility to choose the best suited solution in the “Technology Layer” to realize this service.

To illustrate, it would be perfectly happy to have a charm that implements a http-service and uses a certificate-service to implement secure traffic.
This certificate-service could then be either something fancy like letsencrypt, or just a ‘holder’ for certificate(chain) and private key.
The http-service can be realized with one of the many webservers around, from a python http.server to apache or nginx, just depending on the need.

This would imply that charm have some sort of service naming connected to them so that services can be found, so that optimal choices can be made on what to actually deploy.

17

This is a very interesting proposition. Thanks for sharing! I’m going to start thinking about how this might apply to what I’m working on.

18

I appreciate the sentiment here also.

The kubernetes service config 1:1 mapping through to the charm and the case where the user will want to tune configs outside of what the charm exposes, both seem like counts against this path. Opinionated configuration with limited knobs will allow the charm to be more consumable by novice users, but this often limits how the charm can be used in real life use cases.

Just my 2 cents

1 Like
19

The two approaches don’t seem to be mutually exclusive though. There could be two charms, one called simple-lb with limited knobs and decides for the user which underlying software to use (nginx vs haproxy vs apache) but then there could be an haproxy charm that exposes the full set of configuration options of the software by just allowing the user to provide an haproxy-native configuration file.

In the first example, the charm’s job is to totally abstract out the configuration and operation of the underlying software. The charm user doesn’t even need to care about what software the charm is managing.

In the second instance, the charm’s scope is narrower. It lets the charm user fully configure haproxy and the charm only concerns itself with day 2 operations such as updating, upgrading, backup, migration, etc.

1 Like
20

Just to add, this simplified, “day zero” charm doesn’t justify the overly restrictive schema of config.yaml. Being able to natively support lists or key-value collections may still be useful in this example.

2 Likes