Haproxy + certbot renew + certificates

emcp · 5 February 2022 11:36

I’ve never used mattermost… happy to show you all the charm code… it’s currently published to the charmhub under the beta channel…

Below is an example AWS-ready bundle yaml

if you don’t use spaces, simply drop those bindings… In prod we put the mahrio-webapp behind the private space and NGINX wires towards it via juju relation

If you need more just let me know… I am imagine I would need your gitlab name but happy to get you access… I tried using the official certbot and nginx and sadly it was all just not very clear to me how to use it… my hope was oneday to get off this one-off of mine and wire in the “official” operator… but that is a low low priority in the scheme of things for me…

series: bionic
applications:
  mahrio-certbot:
    charm: mahrio-certbot
    channel: latest/beta
    num_units: 1
    to:
    - "2"
    bindings:
      "": mahrio-dev-public  
      website: mahrio-dev-public
  mahrio:
    charm: mahrio
    channel: latest/beta
    num_units: 1
    to:
    - "1"
    options:
      git_deploy_private_key: |
      ------------------- YOUR PRIVATE KEY HERE -------------------
      mahrio_config: |
        domainnumber2.com
        domainnumber1.com
      mongodb_connection_str: 'GET_FROM_MONGODB_ATLAS'
    bindings:
      "": mahrio-dev-backend
      database: mahrio-dev-backend
      website: mahrio-dev-backend
  mahrio-nginx:
    charm: mahrio-nginx
    channel: latest/beta
    num_units: 1
    to:
    - "2"
    bindings:
      "": mahrio-dev-public
      website: mahrio-dev-public
machines:
  "1":
    constraints: instance-type=t3.small
  "2":
    constraints: instance-type=t3.micro
relations:
- - mahrio:website
  - mahrio-nginx:website
- - mahrio-certbot:website
  - mahrio:website

erik-lonroth · 5 February 2022 22:51

I’ve added a working example to my collection of ops juju charms.

Its not complete yet, but slowly getting there perhaps.

https://github.com/erik78se/juju-operators-examples/blob/main/haproxy-relate/src/charm.py

mthaddon · 7 February 2022 10:00

I think the best way to do this would be via a subordinate charm, and there already seems to be one Deploy Certbot using Charmhub - The Open Operator Collection - the code is https://github.com/canonical/certbot-charm. I haven’t tried it myself, but that’s where I’d start, and the docs include a section on using it with HAProxy.

erik-lonroth · 7 February 2022 10:51

@emcp @joakimnyman this might be worth testing…

Thanx @mthaddon… Too bad Martin Hilton isn’t in the community or maintaining the charm anylonger? Do you know?

jnsgruk · 7 February 2022 11:48

Hey @erik-lonroth - doing some digging internally to see if anyone owns this at the moment. Someone will come back to you (possibly me ;-)) shortly

erik-lonroth · 7 February 2022 13:49

It’s OK, I have a few items open as it seems… the postgresql database backup/restore process with @mthaddon is more of a concern since I have no good way of taking backups with it atm.

The certbot thing is something I think I can manage…

emcp · 7 February 2022 15:42

In fact I chatted with the author of the github that backs this back in beginning of winter… my issue was I couldn’t sort it out and needed to launch “something” … and so just rolled certbot myself… indeed it appears it could offer a nice way forward if I can sort out the plugins it has/uses…

https://github.com/canonical/certbot-charm/issues/13

https://github.com/canonical/certbot-charm/issues/15

and by no means am I knocking the author… I was in a huge rush that month

edit: I recall one reason I didn’t feel too comfortable with this way wasy… I was leaving AWS credentials with the charm and controller… just for the certbot to run once… and opted instead to run a process at the beginning of a new domain launch … something which happens not very often right now… We only use READ-ONLY deploy keys in all our charms … with the exception of course of the controller cloud credential… and we want to keep it that way for as long as possible.

erik-lonroth · 8 February 2022 20:15

@emcp I think Martin Hilton isn’t working for Canonical anylonger and perhaps isn’t maintaining it.

I use DNS acme challenges so I can’t use this charm atm. I’m not sure exactly how I would implement that yet, bkt it would for sure bu useful this kind of charm.

Perhaps we could collaborate on getting this charm updated if you are up for it?

emcp · 9 February 2022 08:17

I think we’re definitely doing the same thing… and I am happy to share knowledge so the best operator charm can arise from whatever work we’ve got… my hope with the Juju Charm community is things like this happening… freedom to experiment on charms myself… while also keeping an eye on who else is working on the same exact modules/components… so that our efforts can be combined… I notice though with some of those older charms, as you mentioned, people move on and then the charmcode is locked in a launchpad or github

with the plugins and subordinate charms I think I still have something to learn how those work and when to use them

alesstimec · 9 February 2022 09:39

Hi,

As i’ve inherited most things from Martin Hilton, let me know if there’s anything related to certbot i can help with…

I can just confirm that we’re using the certbot charm regularly in combination with aws’ route53 and it works a treat. Running the action obtains certs/keys/chain and i quite painless.

Best regards, Ales

erik-lonroth · 9 February 2022 11:17

It seems we can try get something small going together @emcp you and myself.

Would it be possible to have a short meeting some day soon so we can discuss it?

I’m using certbot with dns challenge:

certbot certonly --manual --preferred-challenges dns -d "mysite.example.com" --agree-tos --email foo@example.com

At which point I get a token (**********) which I need to add to my DNS at a TXT record

_acme-challenge.mysite.example.com     ********

I’m not sure how to implement such a solution yet, but perhaps with your help I can figure it out and totally would love to collaborate with you also on how we could discover how to actually improve the community aspects of charming.

Let me know!

alesstimec · 10 February 2022 07:14

Hey Erik,

Of course, feel free to suggest a time and we can shortly discuss needed improvements.

Cheers, Ales

joakimnyman · 11 April 2022 22:11

Hi @alesstimec !

I have tested the certbot charm together with the haproxy charm and it seems to work quite well. However, I can’t figure out how to use domain-alias with it. Do you know if it is supported somehow?

Thanks!

alesstimec · 12 April 2022 05:29

Hey,

Sorry, i don’t think certbot charm supports holding multiple certificates for a unit at the moment. That could certainly be a future improvement.

Cheers, Ales

erik-lonroth · 12 April 2022 05:36

The use-case is that a frontend haproxy (cluster) would serve for multiple domains - such as “unit1.example.com, unit2.example.com, unit3.example.com” etc. I guess.

emcp · 12 April 2022 06:56

Sup Erik,

For nginx it works quite well… we have multiple domains that we instantiate in NGINX… and certbot just auto-speaks to NGINX… prompting you one time which of those domains do you want to use TLS… maybe we can jump on a session later so I can see what HAProxy does differently. The certbot-mahrio charm doesnt track anything about the certs… it just installs certbot and then you have to intervene manually one time… but once that’s done it auto renews / edits NGINX automatically…

erik-lonroth · 12 April 2022 07:33

@joakimnyman @alesstimec and @emcp what about getting a short session together and discuss where we can take this?

We (Dwellir) have a concrete need for this so we are happy to drive the activity forward and supply resources if needed.

joakimnyman · 12 April 2022 19:00

@alesstimec It’s not about holding multiple certificates, I think… Let me explain. We use Route53 and have set up an alias for the dns challenge. Currently we use acme.sh https://github.com/acmesh-official/acme.sh. Using the docker container it would typically be like:

docker run --rm -it -v "$(pwd)/out":/acme.sh -e AWS_ACCESS_KEY_ID=xxx -e AWS_SECRET_ACCESS_KEY=xxx --net=host neilpang/acme.sh --issue --dns dns_aws --domain domain.com --domain-alias acme.domain.dev --email hello@domain.com

So with certbot, the domain for the certificate would be domain.com but for the challenge we want to use acme.domain.dev.

It seems like the charm does only support specifying --domains. And when I think about it, it might be that certbot itself does not support this. If that is the case it would of course be difficult to make the charm support it.

Note that I’m quite new to working with certificates.

Thanks!

joakimnyman · 12 April 2022 20:58

Looks like it has been some work on it a long time ago. https://github.com/certbot/certbot/pull/5350 Which was then moved to the branch https://github.com/certbot/certbot/tree/quinot/topic/dns-follow-cnames

emcp · 13 April 2022 17:08

with our charm for certbot… as of now we run

sudo certbot run

after which you are challenged with the question … which (autodetected via NGINX conf) domains do you want to attempt to retrieve certificates for… including sub-domains… I cannot speak on HAProxy but nginx integration is wonderful

you can see a juju powered webapp at

and my partner with little help got his subdomain wired into TLS as well

And auto-renewal is working as well… so its hands off after that one time execution… I havent looked into further automating say… the Q&A on which domains to enable since… it’s such a low value thing to work on