Migrate from Docker Compose to Juju and Charms

getting started
1

I have a typical Django application stack (Django, Postgres, Celery, Nginx) that is developed and deployed using Docker Compose on VMs with Docker installed. The application is installed with slightly varying configuration on several sites.

The Django Web application including the Celery implementation, for running tasks, is developed privately and their container images are pushed to container image registries that we must authenticate against.

What would it take to turn this setup into something that is managed by Juju?

  • Can we turn this into a set of machine charms?
  • What would a setup look like that allows to also deploy - or (easily) migrate - to Kubernetes charms?

I’m interested in a holistic strategy that would allow us to deploy the application to all of hardware, VMs, PaaS and any public cloud. Is this even possible from a single setup?

1 Like
2

Hi Peter, and welcome!

This kind of cross-cloud deployment is exactly the sort of thing Juju can be used for. Canonical has a team building database charms, for example our PostgreSQL and Kubernetes PostgreSQL charms. At the moment there are significant differences between VM and K8s charms, so we usually develop them separately.

For this, you’d probably use one of our PostgreSQL charms, and perhaps our Nginx Ingress charm for ingress. Then you’d write your Django application charm and integrate the charms using juju integrate to connect them up (and probably use a Juju “bundle”) to allow people to deploy the whole stack together. I don’t believe we have a Celery charm – you could either write your own, or include Celery in the Django application charm.

[Sorry, I accidentally posted early.]

We also have a team working on “12 factor app” tooling, which will help package up Django and similar applications in a charm. I’ll ping David Andersson about that to see if he can give a status update.

So you can definitely turn this into a set of cross-cloud machine charms. Or K8s charms, though again, for now those would probably have to be separate charms. So right now it’s not “easy” to migrate from VMs to K8s.

2 Likes
3

Hi @bittner, as @benhoyt mentioned, we’re working on some tooling to make it easier to manage 12-factor type of apps within juju. We are working towards flask as the first candidate and django as a fast follow. We should have more to say soon including an experimental version of django we would love to get feedback on. Please keep an eye out on this discourse for when we have something to share!

2 Likes
4

Hi Peter

Welcome!

The tooling that @jdkandersson mentions is probably the most interesting thing for you to look at. Essentially, this is a nice way for someone who has an app that can scale out horizontally (doesn’t use local state but rather depends on external databases like Postgres or services like Redis or Kafka) to benefit from the charm ecosystem without themselves having to become a charm expert.

If you dig into that you’ll see there’s a lower layer which handles general issues, and then there is a pluggable upper layer which maps to conventions for specific application development harnesses like Fast API, Flask, Django, Spring Boot etc. At the moment the team is focused on Flask, but it will still give you an idea of how this will feel when they or a Django person build the Django-specific upper layer.

What Juju does nicely is let you consume, as standard parts which you do not have to maintain, the ops code for your external things like databases and message queues. Whether you are in VMs or on K8s, Juju will provide the details to your app to connect to those things, and keep that up to date as those things scale out or failover or get upgraded. Juju does this by having the operational logic of those services encoded as charms, which are like packages of opscode. By using them you’re getting ops as a package, just like apt gives you a kernel as a package :slight_smile:

The challenge of course is that you ALSO need a charm for your app.

What this tooling does is recognise that there are many apps which just need some environment variables set to tell them where those services are. The lower layer does all the work for different kinds of services, and the upper layer bridges that into the specifics of a framework like Django. So, if your app is stateless and uses a framework for which there is such a bridging ‘upper layer’, then you don’t need to write a charm yourself, just use the tooling.

Hope that all makes sense.

As a final comment, for apps which sit nicely inside containers, yes, we are working towards being able to write a single charm which works both on machines and on K8s. A django app would be a good example of that because it probably doesn’t need to trawl about on the host filesystem. Some apps, such as monitoring or storage or other low-level system services, DO need to poke about on the host, so those would stay as what we currently call ‘machine’ charms.

2 Likes
5

Wow, thanks for the elaborate answer, Mark!

Machine charms and system dependencies

That hooks into a question I asked myself lately about machine charms: As an application developer, you containerise your application to solve system dependency issues. You install things like a database client, libraries and development headers that, e.g., your Python modules need in order to get installed. (Also, getting the dependency game right can get quite tricky across Linux distributions, and with a container image you make these dependencies stable.)

Can a machine charm solve this issue? Or do application developers need to stick to containerising, and consequently have to run a Kubernetes flavour on the target host?

System packaging vs. Charms

What I also asked myself is, what is the relationship between the concept of charms and system packaging (.deb, .rpm, etc.)? Why would someone pick developing a machine charm for one’s custom software instead of bundling binaries and deployment procedures and configuration in a system package?

Don’t get me wrong: I love the idea of charms, but I want to be able to answer that question. In the end, both system packages and charms solve ops challenges, don’t they?

6

“getting the thing to run on my system” is step zero of ops, sure, and in that respect a charm is just another package. But there’s more to ops:

  • step one, to (re) configure your thing with high-level knobs so that the user doesn’t need specialized ops knowledge to manage the thing
  • step two, to similarly automatically integrate your thing with other things (and encode the specialized knowledge of how you need to do it), to make it a composable, self-contained, do-one-thing-and-do-it-well piece of a complex puzzle. What’d grep be without pipes? So that the user of your thing doesn’t ALSO need to be an nginx expert to get it to run in production.

These are also addressed by charms, but not by your average deb :slight_smile:

2 Likes