Code sharing in charm monorepos

charm
, ,
1

Recently there have been a few questions about how to share code between charms in a charm monorepo. As far as I’m aware, for now, there are three general approaches that are viable.

  1. Publish the common code as a package and depend on the released version. It can still live in the monorepo. It could even be depended on as a git dependency (if you don’t want to publish to PyPI), though this may get confusing (may want to patch in testing to use latest local version, presumably need to depend on different versions in different channels). This approach doesn’t synergise with the benefits of monorepos (it’s a great fit for polyrepos – one repo per charm + one for the shared code) but the other approaches have their own pain points too, so there may be some appeal. The indirection of depending on a published package instead of on local files has the benefit of making it easy to split the charms into their own separate repos in future. I don’t know if any charm monorepos currently use this approach, but a good example of this approach for polyrepos is mongo-single-kernel-library.

  2. Vendor the common code into the charm directories. Treat the top-level common code as the source of truth, and ensure that the vendored copies are up-to-date in CI. You’ll want to have PRs fail if the code is out of sync, and probably automatically sync the vendored code when running tests, etc. The vault charm monorepo currently uses this approach. The big downside here is the noisy diffs when updating the common code. Also some friction with avoiding failures due to code being out of sync (easy if you use pre-commit or similar though).

  3. If you have a charm directory in your repository, sibling folders aren’t visible to it during charmcraft pack, because charmcraft mounts the project directory when packing. You can currently get around this by running charmcraft pack in the repository root instead. Here’s a draft PR for the vault repo switching to this approach. Note particularly how charmcraft is wrapped in CI, and how the charmcraft.yaml files are changed so they can be interpreted at the repository root, and include the shared code.

You may have heard recommendations to avoid wrapping charmcraft pack, but the current thinking is that 3’s way of wrapping it may be compatible with build processes that invoke charmcraft pack directly by creating repositories for each charm with the appropriate top-level charmcraft.yaml symlink checked in if needed. Methods 1 and 2 don’t wrap charmcraft pack, but it’s not clear how/if they will be compatible with build processes directly invoking charmcraft pack either.

If you’re already using a different approach to code sharing in a charm monorepo, you probably don’t need to change anything at this time, but if you’re thinking of investing significant effort into a different approach it might be worth getting in touch with Charm Tech first (e.g. on matrix).

8 Likes
2

Hey @james-garner,

Thanks for the post !

For solution 1, the development process uses a simple bash script and some test charms (that are a copy-paste of the real charms from the 4 repos): see the script. So when I want to build my charm locally (or during integration tests), I replace the charmcraft pack by bash scripts/build_lib_for_integration.sh and it will build all the charms (or only one if I want so).

This solution is not ideal but it works “good enough” for our usecases for now. I wish however to find a better solution in the future, something closer to 3.

Having migrated most of the integration tests and unit tests in the monorepo, we ensure consistency between the sibling charms (mongodb and mongos) and have an easy release process.

I know at least one more method that I’ve seen used, which is closer to 3 but a bit different. This is sunbeam charms that has a complex build system with custom tooling and yaml files.

5 Likes
3

Hi @gu1nn3zz, thanks for explaining more about the mongo-single-kernel-library approach. It’s good to know how you solved the problem of testing changes to the common code in the charm context, and it’s interesting to hear that you’d like to move closer to a monorepo style approach. I guess that it would be nice to have everything in one place and make updates to the common code and the charms atomically? I think your experience with this approach is an important data point for future discussions about charm monorepos.

Thanks for bringing up the sunbeam charm monorepo too. I’m not super familiar with its infrastructure, but I suspect that it almost certainly goes further with wrapping charmcraft pack than we currently recommend, so I would caution any readers against investing heavily into this approach for new projects right now.

1 Like
4

I just realised I completely forgot to answer to that message!

My goal indeed would be to move all charms in the mongo-single-kernel-library so that all is done atomically, all tests are in one place and all 4 charms are released at the same time. For now we still have that bit of maintenance of opening the 4 PRs (which comes with an extra small risk of forgetting tiny bits of the update like adding a config option / bumping the rock image for kubernetes charms / adding a relation). There are currently some discussions around that project and we’ll see how it goes. If some clearer ideas arise from those discussions, I’ll follow up here!

1 Like
5

hitting this too in the litmus-operators repo. We’re looking now for a solution

1 Like