Haproxy + certbot renew + certificates

Is there anyone who knows how to use Deploy Haproxy using Charmhub - The Open Operator Collection

… in such a way that “certbot renew” could be used?

The charm supports a config “ssl_cert” and “ssl_key”, but thats not any good with “certbot renew” since certbot writes certificate files which can’t easily be added to the charm as far as I know.

I’d love to support development of such a feature, perhaps by allowing for a “certificate-file” option to haproxy, which would then allow for pointing out a certificate which is managed by a “certbot renew” process.

In the latest code, it seems to be possible using the “crts” key, but this documentation is not yet in charmhub…

@mthaddon - I’m working on trying to have certbot doing automatic renewal of certificates. So, I’d be happy to help trying to add support for this.

Is the code maintained in baazar still?

Anyone?

That’s a good question!

I was initially going to suggest writing a subordinate charm, with a “certbot-renew” action. That’s nice and simple, but certbot needs to know a few things about where a web server keeps its files, and the cleanest way to implement that would be a two sided relation, which would require changes to the haproxy charm to support its side of the relation.

Since you’re going to need to alter the charm anyway, it probably makes sense to just add an action directly to the charm.

The launchpad repo is the official home for the code, afaik. If you wanted to submit a patch, I’d suggest doing so, then pinging one of the folks on the haproxy-team. (E.g., Tom Haddon or Cory Johns.)

1 Like

@pengale - I’ve tried out making some bzr merge-requests for @mthaddon to review such as that I can understand how to get changes into the charm. But I also don’t feel comfortable enough to make any changes without first asking how all this can be achieved.

Adding support for letsencrypt certs would be a killer and I hate to have to add this manually as it is today.

1 Like

@pengale @mthaddon

I have been experimenting with this charm that relates to haproxy to get some feeling on it.

https://github.com/erik78se/charm-microsample

I don’t know about HA-Proxy but when I combined NGINX and LetsEncrypts certbot tool… the autorenewal kicked in once the DNS setup was complete… so unfortunately it’s still manual at the very beginning of a deployment… but it’s “fire and forget” afterwards… certbot speaks nginx so it’s auto sees my domains… I simply had to write the configs in my nginx charm

I wrote a website-relation-* set of hooks , here’s the website-relation-changed hook for nginx… webapps will announce themselves via relation alongside their domain(s) , ip, etc…

#!/bin/bash

juju-log -l "WARNING" "(website-relations-changed) starting change"
status-set active "(website-relations-changed)  starting change $(date +"%H:%M")"

# Get the http settings; if not set, ignore
host=`relation-get private-address`
dns_name="$(relation-get dns_name)"
dns_provider="$(relation-get dns_provider)"
node_server_port="$(relation-get node_server_port)"
mahrio_config="$(relation-get mahrio_config)"

if [ -z "$host" ] ; then
    exit 0 # wait for future handshake from http service unit
fi

juju-log -l "WARNING" "(website-relations-changed) host = $host"
juju-log -l "WARNING" "(website-relations-changed) dns_name = $dns_name"
juju-log -l "WARNING" "(website-relations-changed) dns_provider = $dns_provider"
juju-log -l "WARNING" "(website-relations-changed) node-server-port = $node_server_port"
juju-log -l "WARNING" "(website-relations-changed) mahrio_config = $mahrio_config"

while IFS= read -r line 
do
  if [ -z $line ]; then
    echo "skip"
  else
    juju-log -l "WARNING" "(website-relations-changed) line = $line"
    echo -e 'server {' > "/etc/nginx/conf.d/$line.conf" 
    echo -e '\tlisten 80;' >> "/etc/nginx/conf.d/$line.conf"
    echo -e "\tserver_name $line;" >> "/etc/nginx/conf.d/$line.conf"  
    echo -e '' >> "/etc/nginx/conf.d/$line.conf"
    echo -e '\tlocation / {' >> "/etc/nginx/conf.d/$line.conf"
    echo -e '\t\tproxy_set_header   X-Forwarded-For $remote_addr;' >> "/etc/nginx/conf.d/$line.conf" 
    echo -e '\t\tproxy_set_header   Host $http_host;' >> "/etc/nginx/conf.d/$line.conf" 
    echo -e "\t\tproxy_pass         http://$host:$node_server_port;" >> "/etc/nginx/conf.d/$line.conf"
    echo -e '\t}\n}' >> "/etc/nginx/conf.d/$line.conf"  
  fi
done <<< "$mahrio_config"

juju-log -l "WARNING" "(website-relations-changed) restarting nginx after config update"
status-set active "(website-relations-changed)  restarting nginx after config update $(date +"%H:%M")"
systemctl restart nginx
juju-log -l "WARNING" "(website-relations-changed) done restarting nginx after config update"
status-set active "(website-relations-changed) done restarting nginx after config update $(date +"%H:%M")"

juju-log -l "WARNING" "(website-relations-changed) finished change"
status-set active "(website-relations-changed)  finished change $(date +"%H:%M")"

Hope this helps, but I am guessing you’re well aware of this and the issue is HA-Proxy integration is not as smooth?

1 Like

@emcp this looks amazing really.

I’m really interested in what interface you use and the keys you send over the relation to nginx…

host=`relation-get private-address`
dns_name="$(relation-get dns_name)"
dns_provider="$(relation-get dns_provider)"
node_server_port="$(relation-get node_server_port)"
mahrio_config="$(relation-get mahrio_config)"

I’m sure I get those keys from the haproxy.

1 Like

@emcp are you on mattermost atm? I’m working on my charm atm and possibly your nginx setup would be a better choise for me.

Which nginx charm are you using?

I’ve never used mattermost… happy to show you all the charm code… it’s currently published to the charmhub under the beta channel…

Below is an example AWS-ready bundle yaml

if you don’t use spaces, simply drop those bindings… In prod we put the mahrio-webapp behind the private space and NGINX wires towards it via juju relation

If you need more just let me know… I am imagine I would need your gitlab name but happy to get you access… I tried using the official certbot and nginx and sadly it was all just not very clear to me how to use it… my hope was oneday to get off this one-off of mine and wire in the “official” operator… but that is a low low priority in the scheme of things for me…

series: bionic
applications:
  mahrio-certbot:
    charm: mahrio-certbot
    channel: latest/beta
    num_units: 1
    to:
    - "2"
    bindings:
      "": mahrio-dev-public  
      website: mahrio-dev-public
  mahrio:
    charm: mahrio
    channel: latest/beta
    num_units: 1
    to:
    - "1"
    options:
      git_deploy_private_key: |
      ------------------- YOUR PRIVATE KEY HERE -------------------
      mahrio_config: |
        domainnumber2.com
        domainnumber1.com
      mongodb_connection_str: 'GET_FROM_MONGODB_ATLAS'
    bindings:
      "": mahrio-dev-backend
      database: mahrio-dev-backend
      website: mahrio-dev-backend
  mahrio-nginx:
    charm: mahrio-nginx
    channel: latest/beta
    num_units: 1
    to:
    - "2"
    bindings:
      "": mahrio-dev-public
      website: mahrio-dev-public
machines:
  "1":
    constraints: instance-type=t3.small
  "2":
    constraints: instance-type=t3.micro
relations:
- - mahrio:website
  - mahrio-nginx:website
- - mahrio-certbot:website
  - mahrio:website

I’ve added a working example to my collection of ops juju charms.

Its not complete yet, but slowly getting there perhaps.

https://github.com/erik78se/juju-operators-examples/blob/main/haproxy-relate/src/charm.py

1 Like

I think the best way to do this would be via a subordinate charm, and there already seems to be one Deploy Certbot using Charmhub - The Open Operator Collection - the code is https://github.com/canonical/certbot-charm. I haven’t tried it myself, but that’s where I’d start, and the docs include a section on using it with HAProxy.

1 Like

@emcp @joakimnyman this might be worth testing…

Thanx @mthaddon… Too bad Martin Hilton isn’t in the community or maintaining the charm anylonger? Do you know?

Hey @erik-lonroth - doing some digging internally to see if anyone owns this at the moment. Someone will come back to you (possibly me ;-)) shortly :slight_smile:

2 Likes

It’s OK, I have a few items open as it seems… the postgresql database backup/restore process with @mthaddon is more of a concern since I have no good way of taking backups with it atm.

The certbot thing is something I think I can manage…

1 Like

In fact I chatted with the author of the github that backs this back in beginning of winter… my issue was I couldn’t sort it out and needed to launch “something” … and so just rolled certbot myself… indeed it appears it could offer a nice way forward if I can sort out the plugins it has/uses…

https://github.com/canonical/certbot-charm/issues/13

https://github.com/canonical/certbot-charm/issues/15

and by no means am I knocking the author… I was in a huge rush that month :frowning:

edit: I recall one reason I didn’t feel too comfortable with this way wasy… I was leaving AWS credentials with the charm and controller… just for the certbot to run once… and opted instead to run a process at the beginning of a new domain launch … something which happens not very often right now… We only use READ-ONLY deploy keys in all our charms … with the exception of course of the controller cloud credential… and we want to keep it that way for as long as possible.

1 Like

@emcp I think Martin Hilton isn’t working for Canonical anylonger and perhaps isn’t maintaining it.

I use DNS acme challenges so I can’t use this charm atm. I’m not sure exactly how I would implement that yet, bkt it would for sure bu useful this kind of charm.

Perhaps we could collaborate on getting this charm updated if you are up for it?

1 Like

I think we’re definitely doing the same thing… and I am happy to share knowledge so the best operator charm can arise from whatever work we’ve got… my hope with the Juju Charm community is things like this happening… freedom to experiment on charms myself… while also keeping an eye on who else is working on the same exact modules/components… so that our efforts can be combined… I notice though with some of those older charms, as you mentioned, people move on and then the charmcode is locked in a launchpad or github

with the plugins and subordinate charms I think I still have something to learn how those work and when to use them

Hi,

As i’ve inherited most things from Martin Hilton, let me know if there’s anything related to certbot i can help with…

I can just confirm that we’re using the certbot charm regularly in combination with aws’ route53 and it works a treat. Running the action obtains certs/keys/chain and i quite painless.

Best regards, Ales

2 Likes

It seems we can try get something small going together @emcp you and myself.

Would it be possible to have a short meeting some day soon so we can discuss it?

I’m using certbot with dns challenge:

certbot certonly --manual --preferred-challenges dns -d "mysite.example.com" --agree-tos --email foo@example.com

At which point I get a token (**********) which I need to add to my DNS at a TXT record

_acme-challenge.mysite.example.com     ********

I’m not sure how to implement such a solution yet, but perhaps with your help I can figure it out and totally would love to collaborate with you also on how we could discover how to actually improve the community aspects of charming.

Let me know!

Hey Erik,

Of course, feel free to suggest a time and we can shortly discuss needed improvements.

Cheers, Ales

1 Like

Hi @alesstimec !

I have tested the certbot charm together with the haproxy charm and it seems to work quite well. However, I can’t figure out how to use domain-alias with it. Do you know if it is supported somehow?

Thanks!