Shoutout to the charmcraft team for adding the cache sharing mechanism, @syu-w for adding a forward-thinking option to configure the cache directory, and @lengau for your work implementing the feature, help with debugging & brainstorming, and quick bug fixes! charmcraftcache is built on their shoulders & only possible because of their hard work
Thanks @carlcsaposs, this is really neat! I love how it helps solve the first-build UX issues with charmcraft
Am I right in understanding that the cache here is being populated by making a cache from the dependencies of some common charms? This is a great first step and gives that nice pip install charmcraftcache UX, but I wonder if a slight refactor could help it be more tailored to individual users.
Is there a way I can use this tool, or a slightly refactored version, to make a personal cache? I’m picturing something like maybe a Github workflow that could run weekly that looks at my repo and installs all the dependencies and stores that cache blob as an artifact or similar. Then my CI can pull from that store rather than a generic one from pip, and I as a dev own the entire build process of my dependencies but still get the cache hits I’m looking for.
@ca-scribner That’s exactly what it’s designed for—see ccc add
Currently, all the charms share a single cache (since, at the time of development, I didn’t see any reason to split it & it’s more efficient) but theoretically it would be possible to split charmcraftcache-hub into multiple repos. Also, the cache on your host machine is shared, so splitting the caches would require some extra effort to reconcile that back into one cache.
Thanks @carlcsaposs, maybe I misunderstood when quickly reading it the first time.
What I think I’m reading is that ccc add adds my charm to a global list, and ccc pack pulls a global list into my local cache - is that right? If so, I like where its going but wonder if it makes sense that everyone share a cache. What I think a security and efficiency minded user would really like is to maintain their own list of charms+cache built off those, and ccc pack give them their safe wheels rather than wheels coming from a central location. The main idea around requiring charms to have their own wheels was to keep public wheels out of charms.
I think if the personal use case was the focus here, people can get nearly the same benefit but maintain the other procedures already in place. The ccc could still have some default wheels if it wanted (ccc pack --defaults?)
From my perspective, it seems like the main security concern that led to charmcraft using --no-binary was that PyPI distributes binary packages that are uploaded by the package maintainers and that it’s possible to upload a wheel that was not built from the publicly released source code. By building packages from source code on Canonical managed runners, we mitigate this primary security risk.
Regarding other security risks, charmcraftcache-hub is limited to canonical repos to protect against arbitrary code execution (since all python packages can execute arbitrary code during installation). More details: https://github.com/canonical/charmcraftcache/issues/2
One possible mitigation for this would be to build each wheel on a separate runner, but this could get complicated—especially when a package depends on other python packages during its build
I think the current approach strikes a reasonable balance between security and convenience—individual charms don’t need to maintain their own infrastructure (i.e. charmcraftcache-hub & the GH runner); infrastructure can be shared across charms within an organization
There’s certainly room for improvement and, if this tool is useful for charmers, hopefully more development effort can be spent towards the ideas/goals you mentioned
@ca-scribner If you’re interested in a more visual explanation of how charmcraftcache works, I’d recommend trying it out on a charm with ccc add (or pick one of these charms) and ccc pack -v
