How to lock dependencies in Ops framework with charmcraft?

charmcraft
#1

Hello All,

I’m Felipe, from the OpenStack team, we have been asking ourselves the question of how to build reproducible charms with the Ops framework, and we haven’t been able to reach to an answer, and I’m reaching out for feedback :slight_smile:

I was reading the charm_builder.py script, where the CharmBuilder._install_dependencies() suggests there is no option to pass a constraints file, or maybe generate (and use) a lock file from the installed dependencies.

The charm-tools program used to build reactive charms can use a build.lock that describes layers, interfaces and python packages that will be used (if present) to build it, this is an example of it from the ocatavia charm https://opendev.org/openstack/charm-octavia/src/branch/stable/zed/src/build.lock

Is there any best practice or guideline that we could adhere to?

Thanks,

1 Like
#2

Hi :slight_smile:

I guess this kind of depends what you mean. The ops package itself only has two dependencies (PyYAML and python-websockets).

You could easily use pip freeze or similar to generate a requirements.txt with exact pinned versions, and the source code for libraries should be checked into your repo already.

What else do you need to pin within the charm itself other than the Python deps?

Thanks Jon

#3

Hi Jon,

pip freeze would need to be run within the virtualenv created by charmcraft (within the lxd container), which it’s doable, but it certainly requires many steps to get it, here it’s an example:

$ charmcraft pack --shell
Packing the charm                                                                                                                                                                                            
Packing the charm.
root@charmcraft-ironic-dashboard-9721421-0-0-amd64:~/project# cd ../parts/charm/build/
root@charmcraft-ironic-dashboard-9721421-0-0-amd64:~/parts/charm/build# source staging-venv/bin/activate
(staging-venv) root@charmcraft-ironic-dashboard-9721421-0-0-amd64:~/parts/charm/build# pip freeze > /root/requirements.txt
(staging-venv) root@charmcraft-ironic-dashboard-9721421-0-0-amd64:~/parts/charm/build# 
exit
Charms packed:                                                                                                                                                                                               
    ironic-dashboard_ubuntu-22.04-amd64-s390x-ppc64el-arm64_ubuntu-22.10-amd64-s390x-ppc64el-arm64.charm
charm-ironic-dashboard $ lxc start --project charmcraft charmcraft-ironic-dashboard-9721421-0-0-amd64
charm-ironic-dashboard $ lxc file pull --project charmcraft charmcraft-ironic-dashboard-9721421-0-0-amd64/root/requirements.txt ./

These are my questions:

  • Would this be the approach that should be taken?
  • Is there a way to know the name of the container that charmcraft used to do the build?
  • Is passing a constraints file to the build something that has been considered?, I could file a bug to discuss the details of that feature.
#4

I’m not sure I totally agree – the other way of looking at this is that you control the virtualenv being created in the LXD container by supplying a requirements.txt file with all of the dependency tree pinned.

The only difference that could occur is where the patch version of the Python runtime increments in the base that charmcraft uses, but that shouldn’t affect the virtual environment.

If you run charmcraft pack --verbose you’ll see the name of the container being used in the log output. You can also do lxc list --project charmcraft to see the various containers that have been created by charmcraft.

1 Like
#5

you control the virtualenv being created in the LXD container by supplying a requirements.txt file with all of the dependency tree pinned.

Yes, this is definitely how I’d recommend doing it. Commit the pip freezed requirements.txt file to your charm repo and let charmcraft pack use that.

For example, from your charm dir:

$ python3 -m venv venv
$ source venv/bin/activate
$ pip install ops==2.1.1
Collecting ops==2.1.1
...
Successfully installed PyYAML-6.0 ops-2.1.1 websocket-client-1.5.1
$ pip install python-dateutil
Collecting python-dateutil
...
Successfully installed python-dateutil-2.8.2 six-1.16.0
$ pip freeze | tee requirements.txt 
ops==2.1.1
python-dateutil==2.8.2
PyYAML==6.0
six==1.16.0
websocket-client==1.5.1
$ git add requirements.txt
$ git commit -m 'Update deps'
$ charmcraft pack
...