How to manage charm libraries

facundo · 11 January 2021 16:07

See also: Library, Popular charm library index

mthaddon · 11 January 2021 16:23

Does this require a particular version of charmcraft? I have latest/beta (currently 0.6.1) and it doesn’t seem to support the list-lib command.

cory_fu · 11 January 2021 16:50

Does this require a particular version of charmcraft?

I was told:

you need the 0.6.1+76.gcebe5a1 versions, from the edge channel

cory_fu · 11 January 2021 18:06

I’m struggling to understand the reasoning behind tying a library to a specific charm. The only case in which I can see that making any sense is for an interface library where only one charm acts as the provider, but that is a very limited use-case.

For example, I’m currently working on a loadbalancer interface library which would be provided by every one of the cloud integrator charms (OpenStack, AWS, GCP, Azure, vSphere) as well as the kubeapi-load-balancer charm for CK, and possibly others. Under which of those charms would I publish the library? How would a consumer think to look under, e.g., the aws-integrator charm if their expected use-case in on OpenStack?

And what about libraries that aren’t related to an interface and thus aren’t charm specific at all? For example, until Juju has proper support for secrets, all Kubernetes Operator charms could benefit from a library which makes it easy to create and read Secrets in the cluster in order to provide things like passwords for applications.

I know there’s nothing stopping me from publishing my libraries are regular Python packages if that makes more sense for the library in question, but that just means I’m less likely to use this feature and could lead to confusion over where to find any given library.

cory_fu · 11 January 2021 18:21

You say that “there is no need to create further structures to distribute and install the library” but isn’t that exactly what you’re doing here? I can understand the desire to have the libraries more tightly integrated with Charmhub and to keep them separate from / not pollute the general PyPI set of packages, but pip already includes support for an --extra-index-url option which could be managed automatically by charmcraft and the protocol for hosting a PyPI compatible repository is pretty simple and can even be done with a basic web server. So it seems like this could be integrated pretty easily with Charmhub in advanced ways which still use the standard Python conventions (e.g., requirements.txt) within the charm itself.

I can also understand that the packaging semantics and tooling around packaging in the Python ecosystem is fragmented and confusing at best, but you’re also already providing a command to initialize and build libraries, so that seems like a place our tooling could enforce some standards and consistency as well, and potentially provide defaults which simplify the library creation process to avoid the need for most authors to have to deal with the confusion around things like setup.py and pyproject.toml but still have something like that as an option to fall back to when needed.

chris.sanders · 11 January 2021 18:36

This is a really good question. Is the intention to remove ‘generic’ relations from charms altogether? That’s what this appears to be, making interfaces and libraries charm specific and removing all generic interfaces. I liked the concept of generic interfaces that’s how you can have something like a “Loadbalancer” which can be implemented by several back-ends making charms significantly more reusable.

chris.sanders · 11 January 2021 19:00

As I’m working through this workflow a few other difficulties I see.

Since the library lives in a ‘primary’ charm, and is revision controlled with it but released in an independent process:

Those that import a library will also have the library in their version control, changes to libraries will be very confusing on MR’s as they’ll look like code changes.
Bug tracking and feature planning for a library or interface is now mixed with the original charm, where does one file a bug against a charm that has an error in a library?
It’s entirely possible to ship a charm with a version of a library that’s not available on the charm store. This actually goes for both the ‘source’ charm who could forget to publish, and the ‘consumer’ charm who can actually modify the library in place.
I see no good way to fork a library for development and submit the result upstream. IE I can’t install a development branch of a library and use that in my charm until it’s accepted upstream because again the file is simply copied into my source tree bypassing gits knowledge of it being imported at all.

knkski · 11 January 2021 19:01

I would really, really like charmcraft to build on top of existing Python tooling wherever possible, as a charm author. We should have very good, explicit, written-down reasons for going a different route.

To that end, I’ll plug a couple of github issues I’ve opened about integration with existing standards:

https://github.com/canonical/charmcraft/issues/18
https://github.com/canonical/charmcraft/issues/19
https://github.com/canonical/charmcraft/issues/20

facundo · 11 January 2021 19:52

The main use case is exactly as you said, for interface libraries, where the interface is tightly coupled with the charm code.

If it makes sense to provide an interface component in your charm, that interface can be separated as a charm library and published/fetched with this mechanism. If the library doesn’t belong to that charm, but it’s a more generic thing, it’s fine to use other distribution mechanisms.

facundo · 11 January 2021 19:53

The intention is not to remove generic relations from charms.

facundo · 11 January 2021 20:02

As the provided library is tightly coupled with the charm offering the interface, and it lives in that same project, is natural to use the same bug tracking and feature planning.

If you want to use separate bug trackers, or if you have more complex needs for the library development, you can always use other mechanisms.

Regarding forgetting to publish or update the library, we have plans for charmcraft to alert you.

chris.sanders · 11 January 2021 20:34

I’m really glad to hear that. So in essence, you can package things in two way through normal python packaging or via charmcraft libraries.

Does that mean that using a python packaging method is considered a first class citizen of the tooling then? For example, do I still get to use interface versions if I use a standard python package and dependency tracking?

facundo · 12 January 2021 11:30

You can use whatever mechanism you prefer, you have total freedom here.

If you use Charm Libraries, though, it’s just more simple (you want to share just a file, no need to create a project in Github or add setup.py or other mechanisms to make that lib Python-instalable), and your library will be integrated with Charmhub (so other users exploring your charm will find the library as a nice way to interact with it).

cory_fu · 12 January 2021 17:06

My concern with that is fragmentation, especially with discovery. If we’re using two different distribution methods and only a subset are visible on the charm store, it’s going to make things confusing for users.

If a library really does make sense to be tracked within the repo of a specific charm, it can be done so and still use normal Python packaging semantics; all it would require is including a setup.py or pyproject.toml in the lib/ subdirectory. And while I understand the desire for simplicity for a new charm author, you already have a charmcraft create-lib command which creates the boilerplate library files and that could just as easily include a small initial pyproject.toml file.

As I said previously, I well understand the difficulty and confusion around Python packaging, especially in the past, moving away from it means losing a lot of existing features, such as dependency management (what if your lib needs another helper library to function?), package data / resources, entry points, etc. It also risks confusing those who are already familiar with Python, which we’ve established as a requirement. And if we have tooling in place to help create and manage charm-related libraries anyway, we can use that to smooth over rough edges and make opinionated decisions.

This means that library updates can potentially lead to large, noisy PRs against all the charms using the library.

The more I think about it, the more I prefer the idea of Charmhub having an embedded PyPA for charm-related Python packages. The packages could be presented by the Charmhub in the same way that it would for these libraries, but having a single library type / packaging would avoid the issue of having libraries which have more complex requirements not being discoverable at all. It would also allow Charmhub to use the same UI to show what non-charm-related libraries a given charm is using. (There’s also potentially some overlap here with UA-for-apps.)

cory_fu · 18 January 2021 14:43

I’d just like to clarify, since upon re-reading I think it seems that my criticism came across more than my support. I’m very much in favor of the core ideas behind this, namely: simplifying the process of creating, publishing, and consuming libraries for charms, particularly where the standard Python packaging process is overly complex and hard to get started with, and I even like the idea of being able to clearly associate a library with a particular charm when it makes sense. My main concerns basically boil down, then, to ensuring that we can do the same thing for all of the libraries a charm author might want to create or use without fragmenting how those cases are treated.

dylanstathis · 16 February 2021 14:54

Do we have any support for major.minor.patch versioning?

facundo · 18 February 2021 13:37

No, the idea is to explicitly move away from that semantic versioning, that’s why the names are different.

You have the API version which is the one that express your commitment to not break usage (if you do, you have to increase it), so the user can always stay inside the same API version and be safe, and the PATCH version which you increase on every change (and the user will always update to the last PATCH automatically… inside the same API, of course).

jose · 19 March 2021 14:44

Hola @facundo !

I’m following the steps to create a library in the charm I’m working, but I’m getting the following:

$ charmcraft create-lib mylib
Store failure! Name mysql-operator not found in the charm namespace [code: resource-not-found] (full execution logs in /home/jose/snap/charmcraft/common/charmcraft-log-3i6ehv67)

And the log…

$ cat ~/snap/charmcraft/common/charmcraft-log-3i6ehv67 
2021-03-19 11:38:35,706  charmcraft.guard               DEBUG    Starting charmcraft version 0.9.0
2021-03-19 11:38:35,706  charmcraft.main                DEBUG    Raw pre-parsed sysargs: args={'help': False, 'verbose': False, 'quiet': False, 'project_dir': None} filtered=['create-lib', 'mylib']
2021-03-19 11:38:35,707  charmcraft.commands            DEBUG    Couldn't find config file /home/jose/trabajos/canonical/repos/mysql-operator/charmcraft.yaml
2021-03-19 11:38:35,707  charmcraft.main                DEBUG    General parsed sysargs: command='create-lib' args=['mylib']
2021-03-19 11:38:35,708  charmcraft.main                DEBUG    Command parsed sysargs: Namespace(name='mylib')
2021-03-19 11:38:35,712  charmcraft.commands.store      DEBUG    Hitting the store: POST https://api.charmhub.io/v1/charm/libraries/mysql-operator {'library-name': 'mylib'}
2021-03-19 11:38:35,712  charmcraft.commands.store      DEBUG    Loading credentials from file: '/home/jose/snap/charmcraft/common/config/charmcraft.credentials'
2021-03-19 11:38:36,709  charmcraft                     ERROR    Store failure! Name mysql-operator not found in the charm namespace [code: resource-not-found] (full execution logs in /home/jose/snap/charmcraft/common/charmcraft-log-3i6ehv67)

The charmcraft init command didn’t create the charmcraft.yaml file when I started the charm some weeks ago.

As far as I understand the problem seem to be that the charm I am working on (mysql-operator) it’s not yet in charmhub (because it’s a work-in-progress).

If so, is there a way to get the lib template if the charm it’s not published yet?

facundo · 23 March 2021 19:16

Hola José,

To create a library you need for the charm to be registered in Charmhub, yes, because the server assigns an id to the library.

This id and the whole structure is for you to share the library easily.

When starting its development (that for sure it’s not ready to publish it as a library, as everything is still being bootstrapped), you can have that library in any file together with your charm and just use it. Then, when the initial versions of the charm are in Charmhub, create the library, fill it with the code you developed and tuned elsewhere, and publish it.

Hope this help you!

pedroleaoc · 7 April 2022 09:25