I’ve been dealing with building rocks lately and I wanted to share my process, mostly when debugging and fixing things I thought would work and then they didn’t. 
the tl;dr
For the very impatient, this is what I cover.
- check that iptables is not blocking your LXC instance and allow traffic on lxdbr0
- use --debug to shell into a failing step and --verbosity debug to show more output
- use rockcraft --debug --verbosity debug to execute a step over a part for more fine-grained control and reducing the troubleshoting space
- if a rockcraft clean fails, delete the LXC instances under the rockcraft project
- check some existing rockcrafts under Canonical or elsewhere for reference
rockcraft intro
rockcraft is a tool for building OCI-compliant images (and has a great doc page). Coming from docker, you might find some differences that are worth noting. Also, rockcraft gives you more power to debug your container declaration (a file named rockcraft.yaml, as opposed to the Dockerfile that is pretty well known).
This is a sample of some things I always check and do when building or refactoring rocks. Note that I assume you have rockcraft already installed (if you don’t please refer to the documentation and you’ll be set in no time).
docker and iptables
If you happen to have timeouts in your LXC instance that is used for building the rock, chances are you have a networking issue that is caused by docker and the associated iptables rules.
sudo iptables -nL # list rules (look for DOCKER-USER)
sudo iptables -I DOCKER-USER -i lxdbr0 -j ACCEPT # allow lxdbr0 interface for incomming traffic
sudo iptables -I DOCKER-USER -o lxdbr0 -j ACCEPT # do the same for outgoing traffic
With that in place, you rockcraft commands should succeed.
a shell for failing builds
The approach with docker is usually that you create a Dockerfile, make changes to it, build and if it fails - you make changes again, build and iterate until you get to a clean build. We can do better than that with rockcraft.
Rockcraft gives you the option to shell into the LXC instance buildind your OCI image, this can happen before, after or on failure during any step.
I usually have this alias
alias rpd="rockcraft pack --debug --verbosity debug"
and this gives me a shell in case the pack fails and I get to see all the gory details during the pack phase.
The debug verbosity gives you enough information that you can try and resolve the issue within the IXC instance and then implement the fix in the rockcraft.yaml file later. It is a powerful feature that can save some time.
moving confidently through complex rockfiles
Container images can get quite complex with lots of dependencies and moving parts. Initially you might think that the idea of separating everything into parts and the pack process into multiple steps adds to that complexity, though there is also more power and flexibility given to the engineer building the rockfile when they try to understand what is happening.
The “what is happening” moment accounting for 80% of the time spent in Engineering, having more power is not so bad.
You can control the process of the rockraft pack with a very fine comb, going gradually through the lifecycle of the OCI image (pull, overlay, build, stage, prime) and even targeting specific parts of your rockfile in a specific step.
Say I have a rockcraft.yaml file
# Copyright 2023 Canonical Ltd.
# See LICENSE file for licensing details.
name: foobar
summary: Foobar rock
description: Foobar OCI image for the Foobar charm
base: ubuntu@20.04
run-user: _daemon_
license: Apache-2.0
version: "1.0"
platforms:
amd64:
parts:
node-tools:
plugin: nil
source: .
overlay-script: |
curl -sL https://deb.nodesource.com/setup_18.x | bash -
apt-get -y install nodejs
npm install -g yarn
runtime-packages:
plugin: nil
overlay-packages:
- git
- libyaml-dev
foobar:
plugin: dump
after:
- node-tools
- runtime-packages
source: https://github.com/foo/bar.git
source-depth: 1
source-type: git
override-build: |
craftctl default
touch log/production.log
yarn install --immutable
organize:
"*": srv/foobar/app/
rad-plugin:
plugin: dump
source: https://github.com/foo/rad-plugin.git
source-depth: 1
after:
- foobar
organize:
"*": srv/foobar/app/plugins/rad-plugin/
gems:
plugin: nil
after:
- foobar
- rad-plugin
build-packages:
- libpq-dev
- git
- libssl-dev
- libyaml-dev
- ubuntu-dev-tools
build-snaps:
- ruby/3.2/stable # if you google "ruby snap", you get taken to a chocolate chip cookie website
override-build: |
cd $CRAFT_STAGE/srv/foobar/app
gem install -n "bin" bundler
bin/bundle install
organize:
'/var/lib/gems': var/lib/gems
'srv/foobar/app/bin/*': srv/foobar/app/bin/
'srv/foobar/app/bundle': srv/foobar/app/
You have a couple of parts that add up to a Rails application with some Javascript sprinkled on top 
. If I’m certain about some parts (installing packages via apt-get and npm should be normally fine), I might want to dive into the actual application that I want to deploy, the glorious foobar Ruby on Rails! 
rockcraft build foobar --debug --verbosity debug
Notice that the foobar part comes after nodejs and some runtime deps, so they will get pulled in by rockcraft before doing the build for the foobar part. Having the --verbosity debug will also show you some nice progress. If everything is fine, you just halved your problem space.
A part can also fail in some unexpected way due to dependencies on other parts and misconfigurations on the overrides across steps.
An interesting bug I had was that a part was failing, though when I get dropped in the shell via the --debug flag of rockcraft, executing the same commands actually worked. So I started with a rockcraft build that passed, and moving on to rockcraft stage it failed - and that pointed me in the direction of a path being overridden in the staging area, causing a failure.
when all else fails
You can clean parts with rockcraft clean part and you can clean the whole project with rockcraft clean. In case for some reason this doesn’t do the trick and you start suspecting rockcraft instead of doubting yourself, behind the scenes rockcraft uses LXD and two LXC instances in a project space rockcraft.
lxc list --project rockcraft # has rockcraft-* and base-instance-* instances
lxc delete <rockcraft-*> --project rockcraft # deletes the rockcraft-* instance
Perhaps you shouldn’t go as deep, though you might want to know that this could be an option as well.
getting inspiration
There are a whole lot of good examples on how to write rockfiles these days and I need to mention that you can get a lot of reference for free. Searching for rockcraft.yaml just under the Canonical space on Github yields a lot of good examples, ranging from basic to some pretty complex stuff. You can see howto mix apt packages and snaps, how to override specific steps during the lifecycle, setting environment variables and making use of the ones that are part of the craft idea (because tools like snapcraft make use of these as well for instance).
the end
This is not an exhaustive list, though I hoped you enjoyed reading it and it can also save you some time on your next “what is happening” journey.
If you have tips and tricks when working with rockcraft, please share them in the comment section! 