Python dependencies in Charm Libraries

charmcraft
1

See the docs: Library > Metadata > PYDEPS

Charm libraries often need to use regular Python libraries.

However, they have a very specific structure aimed for simplicity and integration with charms and Charmhub. One of the characteristics (by design) is that they shall have only one Python file, which is a problem the moment the library needs to specify Python dependencies, as in the Python world all mechanisms implies a separate file.

Since charmcraft 2.2.0+89.g8a3a0e6 (currently released in the edge channel) there is a mechanism to declare in the charm library’s unique file which dependencies are needed by that library: the PYDEPS key.

This is a new special key in the charm library (the others are LIBID, LIBAPI and LIBPATCH). Unlike those previous keys, PYDEPS is optional and it’s not a single value but a list of strings, each string being a regular “pip installable” Python dependency that will be normally retrieved from PyPI (but subject to user’s system configuration), and supporting all formats (just the package name, a link to a Github project, etc.).

The following are examples of possible PYDEPS declarations.

PYDEPS=["jinja2"]

PYDEPS = ["pyyaml", "httpcore<0.15.0,>=0.14.5"]

PYDEPS = [
    "git+https://github.com/canonical/operator/#egg=ops",
    "httpcore<0.15.0,>=0.14.5",
    "requests",
]

Of course only one PYDEPS declaration is allowed, if any.

Charmcraft will collect the dependencies from all libraries included in the charm and install them from source in the virtual environment inside the .charm file, together with all the other Python dependencies declared by the charm itself.

Note that pip, when called to install all the dependencies from the charm and all the used libraries, may detect conflicts between the requested packages and their versions. This is a feature, because it’s always better to detect incompatibilities between dependencies at this moment than when the charm is being deployed or run after deployment.

5 Likes
2

Hello,

Naive question: has using Python packages for charm libraries been considered?

At the level of complexity that comes with adding dependencies to charm libraries, it seems like using Python packages instead of a single Python file might be simpler from the perspective of a charm developer.

With Python packages, dependency management could be compatible with tools in the Python ecosystem like pip-tools, Poetry, and Renovate. This would enable charm developers to use existing tooling, like what the Kubeflow team has implemented with pip-tools.

Also, maybe packages could help alleviate some of the limitations that have shown up from charm libraries being used beyond just the simple case of specifying relation interfaces for a single charm. Here are a few examples:

  • The data-platform-libs charm is not a charm that can be deployed, it only exists to host libraries
  • The mongodb charms use an internal charm library to share code between VM & Kubernetes charms. Python packages could enable much greater code sharing between the charm code for VMs and Kubernetes.
  • The mysql VM & Kubernetes charms both use the mysql library, but only the VM charm “owns” the library
1 Like
3

Hello @carlcsaposs !

You can always package your library and publish it as a regular Python Package in PyPI.

The whole idea of charm libraries is to offer a very simple very straightforward solution for people to share code around their charms, if your needs escape them having the fallback of PyPI is a standard procedure.

Maybe in the future we’ll consider a way for developers to get their PyPI libraries listed in charmhub.io, but no conversations about this yet.

4

Got it, thank you!

Does charmcraft support installing packages locally (e.g. ./foo/ or ../foo/ in requirements.txt)?

I tried it and it appears to work if the package is in the charm directory (./foo/) but not outside the charm directory (../foo/).

However, it appears that changes to the package require a charmcraft clean for them to show up in the .charm file. I’m guessing this means that local packages aren’t officially supported—is that correct?