Charm Engineering is always looking for ways to improve our processes to developing, building, and testing charms. Even conventional tools are being scrutinised, and nothing is considered an ideal tool.
Enter uv, a Rust-based package manager for Python that claims to be 10 to 100 times faster than traditional options ( pip, poetry, …).
Summary of Changes
Directory structure
├── Makefile
├── charmcraft.yaml
├── pyproject.toml
├── uv.lock
├── src/
│ ├── main.py
We no longer need the requirements.txt and tox.ini files.
Test Commands
tox -e lint→make linttox -e static→make statictox -e fmt→make fmttox -e unit→make unittox -e scenario→make scenariotox -e integration→make integration
Do we still need Tox?
For the migration, we will look at what changes are required to implement uv.
Using a Makefile for Setup
We decided to move away from tox because it manages its own virtual environments in .tox whereas, uv defaults to .venv. It felt weird to try and sync them, so instead we went with Make which is acting as our command runner to execute tests and prepare (using uv) environments.
The following is the Makefile portion which handles setup
PROJECT := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
SRC := $(PROJECT)src
TESTS := $(PROJECT)tests
ALL := $(SRC) $(TESTS)
export PYTHONPATH = $(PROJECT):$(PROJECT)/lib:$(SRC)
export PY_COLORS=1
# Update uv.lock with the latest deps
lock:
uv lock --upgrade --no-cache
# Generate requirements.txt from pyproject.toml
requirements:
uv export --frozen --no-hashes --format=requirements-txt -o requirements.txt
make lock- This command updates the
uv.lockfile by using thedependenciessection of thepyproject.tomlwithuvlogic to determine how to achieve an explicit web of pinned dependencies (uv.lock).
- This command updates the
make requirements- This command dynamically creates the
requirements.txtfile during the charmcraft build step from theuv.lockfile.
- This command dynamically creates the
Using a Makefile for Testing
Within the same Makefile we also define our tests:
...
lint: <SOME TESTS>
static: <SOME TESTS>
fmt: <SOME TESTS>
unit: <SOME TESTS>
integration:
uv run --frozen --isolated --extra dev \
pytest \
...
$(TESTS)/integration \
$(ARGS)
The use of the --frozen, --isolated, and --extra options are interesting here. These are specific to uv run which runs (in this example) the pytest command without updating the uv.lock file, running in an isolated virtual environment, and using the dev optional dependencies.
The integration test can also be run with optional arguments:
make integration ARGS="pytest-specific args"
Packing the Charm
Since we no longer have a requirements.txt file in the root of the charm, we need to modify the charmcraft.yaml.
parts:
charm:
build-snaps:
- astral-uv
override-build: |
make requirements
craftctl default
charm-requirements: [requirements.txt]
The charm-requirements defaults to [requirements.txt], but without setting it (counter-intuitive), the charm does not respect the requirements.txt when generated dynamically during build.
We now install uv via the atral-uv snap into the charm and generate the requirement.txt dynamically prior to resuming normal charm build operations with craftctl default.
Defining Dependencies
By upgrading the pyproject.toml with a project.dependencies and project.optional-dependencies sections we can set our Python dependency pins here. The idea is to leave this unpinned as possible to allow uv to determine the combinations of packages to satisfy the requirements.
[project]
name = "my-lightning-fast-charm"
version = "0.0" # this is in fact irrelevant
requires-python = "==3.8.*" # https://packages.ubuntu.com/focal/python3
dependencies = [
"ops",
# ---PYDEPS---
"cosl>=0.0.43",
"pydantic>=2",
]
[project.optional-dependencies]
dev = [
# Linting
"ruff",
# Unit
"pytest",
# Integration
"juju",
"pytest-operator",
]
When deciding which requires-python version to pin to, refer to the build-on key in the charmcraft.yaml file. The Ubuntu release will come with a specific Python version which can be found here for focal.
We currently need to sync the PYDEPS of our charm into the pyproject.toml since the charmcraft.yaml only respects the pyproject.toml.
If your library requires any dependencies outside the standard lib, ops, and the transitive dependencies of ops, then create a regular Python package (and installing from PyPI or from source control or wherever you like) than using a charm lib. However, it’ll be a while before that becomes common practice.
Backwards Compatible CI
Since we are gradually moving away from tox we need a way to support both tox and uv with make in CI. The fix for this is to conditionally check for the existence of the tox.ini file and run the respective testing command.
- name: Install dependencies
run: |
python3 -m pip install tox
sudo snap install --classic astral-uv
- name: Run linters
run: |
cd ${{ inputs.charm-path }}
[ -f tox.ini ] && tox -vve lint || make lint