Creating and using charm libraries

facundo · January 11, 2021, 4:07pm

Charm authors need a way to easily share and reuse logic, charms make that even more important given the two-sided nature of relations. That is, a given interface type needs logic on the providing side and on the required side, this is better handled when the responsibility lies on the same entity.

The Charmcraft tool supports a first-class mechanism to reuse logic in a form of python modules named libraries which are published on Charmhub for easy consumption.

This model diverges from generic versioning systems (as git/Github) and packages publishing systems (like PyPI) because we aim for simplicity as there is no need to create further structures to distribute and install the library, nor we need to have registered users in other platforms.

Furthermore, libraries shared through this mechanism are directly integrated with Charmhub, allowing other users to find our shared libraries (including their documentation) when exploring our charms on that platform.

The structure on disk

Libraries are located in a specific directory inside a charm project with the following structure:

$CHARMDIR/lib/charms/<charm>/v<API>/<libname>.py

$CHARMDIR is the project’s root (contains src/, hooks/, etc.), and the <charm> placeholder represents the charm responsible for the library named as <libname>.py with API version <API>. So, as a more concrete example:

$CHARMDIR/lib/charms/mysql/v3/db.py

In this case, the author of charm mysql is publishing the library db with major version 3. This file may be used both by the author and by any other charm authors that are interested in the offered functionality.

The library may then be imported by its fully qualified path:

import charms.mysql.v3.db

Inside db.py the following fields must be defined:

LIBID = "abcdef1234"
LIBAPI = 3    # Must match the major version in the import path.
LIBPATCH = 2  # The current patch version. Must be updated when changing.

LIBID is a unique identifier for the library across the entire universe of charms. This is assigned by Charmhub to this particular library automatically at library creation time. That id enables Charmhub and charmcraft to track the library uniquely even if the charm or the library are renamed, which allows updates to warn and guide users through the process.

Creating and sharing a charm library

Let’s start from a charm that has no libraries at all:

jdoe@machine:/home/john/blogsystem$ ll lib
ls: cannot access 'lib': No such file or directory

The library we will be creating belongs to a charm, so we will not only be working inside a charm’s directory, but this needs to be registered in Charmhub (later we’ll see that we can ask Charmhub to list all libraries belonging to a given charm).

The first step is create the bare library template:

jdoe@machine:/home/john/blogsystem$ charmcraft create-lib superlib
Library charms.blogsystem.v0.superlib created with id e76db596f4bb44fd9c0ce068669fc2ac.
Consider 'git add lib/charms/blogsystem/v0/superlib.py'.

That command created a file on disk, inside the proper directory structure we mentioned above. It’s a good idea to incorporate this new file now to your code versioning system.

jdoe@machine:/home/john/blogsystem$ ll lib/charms/blogsystem/v0/superlib.py 
-rw-rw-r-- 1 jdoe jdoe 1048 Dec 17 13:46 lib/charms/blogsystem/v0/superlib.py

That file is just a template we must fill, though. We can separate the content in three parts:

the module’s docstring: this will be the library’s documentation, automatically presented on Charmhub and updated whenever a new version of the library is published; markdown is supported, following the CommonMark specification.
the three metadata fields: LIBID is already assigned by Charmhub, never change it; LIBAPI and LIBPATCH are set to an initial state (0 and 1 correspondingly), which is fine for now, below we’ll see how/when to update these.
the rest of the file: here is where we will add all the library’s code.

Now we’re ready to publish our library. This is done automatically with one command, both uploading the library content and making it available for everybody.

jdoe@machine:/home/john/blogsystem$ charmcraft publish-lib charms.blogsystem.v0.superlib
Library charms.blogsystem.v0.superlib sent to the store with version 0.1

After this step, our library is ready to be used by other developers.

We would eventually need to evolve the library’s content (new functionalities, bug fixing, documentation improvements, etc.). Every time we want to offer a new version of our library, we will need to call the publish-lib command.

However, before publishing new versions, we need to update the LIBAPI/LIBPATCH metadata fields inside the library file. Most times it is enough to just increment LIBPATCH, but if we’re introducing breaking changes we must work with the major API version.

jdoe@machine:/home/john/blogsystem$ charmcraft publish-lib charms.blogsystem.v0.superlib
Library charms.blogsystem.v0.superlib sent to the store with version 0.2

We need to take in consideration that users of our library will update it automatically to the latest PATCH version with the same API version.

To avoid breaking other people’s library usage we should increment the LIBPAPI version and reset LIBPATCH to 0. But before adding the breaking changes and updating these values, we should copy the library to the new path:

jdoe@machine:/home/john/blogsystem$ mkdir lib/charms/blogsystem/v1
jdoe@machine:/home/john/blogsystem$ cp lib/charms/blogsystem/v0/superlib.py lib/charms/blogsystem/v1/

This way we can maintain different major API versions independently, being able to update our v0 after we published v1.

jdoe@machine:/home/john/blogsystem$ charmcraft publish-lib charms.blogsystem.v1.superlib
Library charms.blogsystem.v1.superlib sent to the store with version 1.0

Discovering libraries

If the charm we’re writing needs to interact with another service, we should check if that service provides any library that would help in that interaction.

The easiest way to see this from the command line is using charmcraft's list-lib command, which will query Charmhub and show which libraries are published for the specified charm, indicating also the API/patch versions for the corresponding tips.

jdoe@machine:/home/jane/autoblog$ charmcraft list-lib blogsystem
Library name    API    Patch
superlib        1      0

Listing does not show older API versions (above we published superlib also in v0.2), this ensures that new users always start with the latest version.

Another good entry point for searching libraries is to explore The Open Operator Collection.

Using a shared charm library

The moment we decided that we need to use another charm’s library in our charm, we need to fetch it. This is done through the fetch-lib command in the Charmcraft tool:

jdoe@machine:/home/jane/autoblog$ charmcraft fetch-lib charms.blogsystem.v1.superlib
Library charms.blogsystem.v1.superlib version 1.0 downloaded.

That command will download the library itself, creating the required directory layout:

jdoe@machine:/home/jane/autoblog$ ll lib/charms/blogsystem/v1/superlib.py 
-rw-rw-r-- 1 jdoe jdoe 1061 Dec 17 15:24 lib/charms/blogsystem/v1/superlib.py

This file is now part of our charm’s project. It will be packed inside our charm and distributed with it. And all we need to use it is to directly import it from our charm’s code (note that the charm automatically has the lib directory as part of the Python import paths).

from charms.blogsystem.v1 import superlib

In the future we may want to update the library we’re using: all we need to do is run the same fetch-lib command (as we already have the library, it will only update it if necessary):

jdoe@machine:/home/jane/autoblog$ charmcraft fetch-lib charms.blogsystem.v1.superlib
Library charms.blogsystem.v1.superlib was already up to date in version 1.0.

mthaddon · January 11, 2021, 4:23pm

Does this require a particular version of charmcraft? I have latest/beta (currently 0.6.1) and it doesn’t seem to support the list-lib command.

cory_fu · January 11, 2021, 4:51pm

Does this require a particular version of charmcraft?

I was told:

you need the 0.6.1+76.gcebe5a1 versions, from the edge channel

cory_fu · January 11, 2021, 6:06pm

I’m struggling to understand the reasoning behind tying a library to a specific charm. The only case in which I can see that making any sense is for an interface library where only one charm acts as the provider, but that is a very limited use-case.

For example, I’m currently working on a loadbalancer interface library which would be provided by every one of the cloud integrator charms (OpenStack, AWS, GCP, Azure, vSphere) as well as the kubeapi-load-balancer charm for CK, and possibly others. Under which of those charms would I publish the library? How would a consumer think to look under, e.g., the aws-integrator charm if their expected use-case in on OpenStack?

And what about libraries that aren’t related to an interface and thus aren’t charm specific at all? For example, until Juju has proper support for secrets, all Kubernetes Operator charms could benefit from a library which makes it easy to create and read Secrets in the cluster in order to provide things like passwords for applications.

I know there’s nothing stopping me from publishing my libraries are regular Python packages if that makes more sense for the library in question, but that just means I’m less likely to use this feature and could lead to confusion over where to find any given library.

cory_fu · January 11, 2021, 6:23pm

You say that “there is no need to create further structures to distribute and install the library” but isn’t that exactly what you’re doing here? I can understand the desire to have the libraries more tightly integrated with Charmhub and to keep them separate from / not pollute the general PyPI set of packages, but pip already includes support for an --extra-index-url option which could be managed automatically by charmcraft and the protocol for hosting a PyPI compatible repository is pretty simple and can even be done with a basic web server. So it seems like this could be integrated pretty easily with Charmhub in advanced ways which still use the standard Python conventions (e.g., requirements.txt) within the charm itself.

I can also understand that the packaging semantics and tooling around packaging in the Python ecosystem is fragmented and confusing at best, but you’re also already providing a command to initialize and build libraries, so that seems like a place our tooling could enforce some standards and consistency as well, and potentially provide defaults which simplify the library creation process to avoid the need for most authors to have to deal with the confusion around things like setup.py and pyproject.toml but still have something like that as an option to fall back to when needed.

chris.sanders · January 11, 2021, 6:36pm

This is a really good question. Is the intention to remove ‘generic’ relations from charms altogether? That’s what this appears to be, making interfaces and libraries charm specific and removing all generic interfaces. I liked the concept of generic interfaces that’s how you can have something like a “Loadbalancer” which can be implemented by several back-ends making charms significantly more reusable.

chris.sanders · January 11, 2021, 7:00pm

As I’m working through this workflow a few other difficulties I see.

Since the library lives in a ‘primary’ charm, and is revision controlled with it but released in an independent process:

Those that import a library will also have the library in their version control, changes to libraries will be very confusing on MR’s as they’ll look like code changes.
Bug tracking and feature planning for a library or interface is now mixed with the original charm, where does one file a bug against a charm that has an error in a library?
It’s entirely possible to ship a charm with a version of a library that’s not available on the charm store. This actually goes for both the ‘source’ charm who could forget to publish, and the ‘consumer’ charm who can actually modify the library in place.
I see no good way to fork a library for development and submit the result upstream. IE I can’t install a development branch of a library and use that in my charm until it’s accepted upstream because again the file is simply copied into my source tree bypassing gits knowledge of it being imported at all.

knkski · January 11, 2021, 7:01pm

I would really, really like charmcraft to build on top of existing Python tooling wherever possible, as a charm author. We should have very good, explicit, written-down reasons for going a different route.

To that end, I’ll plug a couple of github issues I’ve opened about integration with existing standards:

https://github.com/canonical/charmcraft/issues/18
https://github.com/canonical/charmcraft/issues/19
https://github.com/canonical/charmcraft/issues/20

facundo · January 11, 2021, 7:52pm

The main use case is exactly as you said, for interface libraries, where the interface is tightly coupled with the charm code.

If it makes sense to provide an interface component in your charm, that interface can be separated as a charm library and published/fetched with this mechanism. If the library doesn’t belong to that charm, but it’s a more generic thing, it’s fine to use other distribution mechanisms.

facundo · January 11, 2021, 7:53pm

The intention is not to remove generic relations from charms.

facundo · January 11, 2021, 8:02pm

As the provided library is tightly coupled with the charm offering the interface, and it lives in that same project, is natural to use the same bug tracking and feature planning.

If you want to use separate bug trackers, or if you have more complex needs for the library development, you can always use other mechanisms.

Regarding forgetting to publish or update the library, we have plans for charmcraft to alert you.

chris.sanders · January 11, 2021, 8:34pm

I’m really glad to hear that. So in essence, you can package things in two way through normal python packaging or via charmcraft libraries.

Does that mean that using a python packaging method is considered a first class citizen of the tooling then? For example, do I still get to use interface versions if I use a standard python package and dependency tracking?

facundo · January 12, 2021, 11:30am

You can use whatever mechanism you prefer, you have total freedom here.

If you use Charm Libraries, though, it’s just more simple (you want to share just a file, no need to create a project in Github or add setup.py or other mechanisms to make that lib Python-instalable), and your library will be integrated with Charmhub (so other users exploring your charm will find the library as a nice way to interact with it).

cory_fu · January 12, 2021, 5:06pm

My concern with that is fragmentation, especially with discovery. If we’re using two different distribution methods and only a subset are visible on the charm store, it’s going to make things confusing for users.

If a library really does make sense to be tracked within the repo of a specific charm, it can be done so and still use normal Python packaging semantics; all it would require is including a setup.py or pyproject.toml in the lib/ subdirectory. And while I understand the desire for simplicity for a new charm author, you already have a charmcraft create-lib command which creates the boilerplate library files and that could just as easily include a small initial pyproject.toml file.

As I said previously, I well understand the difficulty and confusion around Python packaging, especially in the past, moving away from it means losing a lot of existing features, such as dependency management (what if your lib needs another helper library to function?), package data / resources, entry points, etc. It also risks confusing those who are already familiar with Python, which we’ve established as a requirement. And if we have tooling in place to help create and manage charm-related libraries anyway, we can use that to smooth over rough edges and make opinionated decisions.

This means that library updates can potentially lead to large, noisy PRs against all the charms using the library.

The more I think about it, the more I prefer the idea of Charmhub having an embedded PyPA for charm-related Python packages. The packages could be presented by the Charmhub in the same way that it would for these libraries, but having a single library type / packaging would avoid the issue of having libraries which have more complex requirements not being discoverable at all. It would also allow Charmhub to use the same UI to show what non-charm-related libraries a given charm is using. (There’s also potentially some overlap here with UA-for-apps.)

cory_fu · January 18, 2021, 2:43pm

I’d just like to clarify, since upon re-reading I think it seems that my criticism came across more than my support. I’m very much in favor of the core ideas behind this, namely: simplifying the process of creating, publishing, and consuming libraries for charms, particularly where the standard Python packaging process is overly complex and hard to get started with, and I even like the idea of being able to clearly associate a library with a particular charm when it makes sense. My main concerns basically boil down, then, to ensuring that we can do the same thing for all of the libraries a charm author might want to create or use without fragmenting how those cases are treated.

dylanstathis · February 16, 2021, 2:54pm

Do we have any support for major.minor.patch versioning?

facundo · February 18, 2021, 1:37pm

No, the idea is to explicitly move away from that semantic versioning, that’s why the names are different.

You have the API version which is the one that express your commitment to not break usage (if you do, you have to increase it), so the user can always stay inside the same API version and be safe, and the PATCH version which you increase on every change (and the user will always update to the last PATCH automatically… inside the same API, of course).

jose · March 19, 2021, 2:44pm

Hola @facundo !

I’m following the steps to create a library in the charm I’m working, but I’m getting the following:

$ charmcraft create-lib mylib
Store failure! Name mysql-operator not found in the charm namespace [code: resource-not-found] (full execution logs in /home/jose/snap/charmcraft/common/charmcraft-log-3i6ehv67)

And the log…

$ cat ~/snap/charmcraft/common/charmcraft-log-3i6ehv67 
2021-03-19 11:38:35,706  charmcraft.guard               DEBUG    Starting charmcraft version 0.9.0
2021-03-19 11:38:35,706  charmcraft.main                DEBUG    Raw pre-parsed sysargs: args={'help': False, 'verbose': False, 'quiet': False, 'project_dir': None} filtered=['create-lib', 'mylib']
2021-03-19 11:38:35,707  charmcraft.commands            DEBUG    Couldn't find config file /home/jose/trabajos/canonical/repos/mysql-operator/charmcraft.yaml
2021-03-19 11:38:35,707  charmcraft.main                DEBUG    General parsed sysargs: command='create-lib' args=['mylib']
2021-03-19 11:38:35,708  charmcraft.main                DEBUG    Command parsed sysargs: Namespace(name='mylib')
2021-03-19 11:38:35,712  charmcraft.commands.store      DEBUG    Hitting the store: POST https://api.charmhub.io/v1/charm/libraries/mysql-operator {'library-name': 'mylib'}
2021-03-19 11:38:35,712  charmcraft.commands.store      DEBUG    Loading credentials from file: '/home/jose/snap/charmcraft/common/config/charmcraft.credentials'
2021-03-19 11:38:36,709  charmcraft                     ERROR    Store failure! Name mysql-operator not found in the charm namespace [code: resource-not-found] (full execution logs in /home/jose/snap/charmcraft/common/charmcraft-log-3i6ehv67)

The charmcraft init command didn’t create the charmcraft.yaml file when I started the charm some weeks ago.

As far as I understand the problem seem to be that the charm I am working on (mysql-operator) it’s not yet in charmhub (because it’s a work-in-progress).

If so, is there a way to get the lib template if the charm it’s not published yet?

facundo · March 23, 2021, 7:17pm

Hola José,

To create a library you need for the charm to be registered in Charmhub, yes, because the server assigns an id to the library.

This id and the whole structure is for you to share the library easily.

When starting its development (that for sure it’s not ready to publish it as a library, as everything is still being bootstrapped), you can have that library in any file together with your charm and just use it. Then, when the initial versions of the charm are in Charmhub, create the library, fill it with the code you developed and tuned elsewhere, and publish it.

Hope this help you!