Karma rock, play-by-play

[sed-i (Leon)]

Karma is a very handy UI for alertmanager, and it was charmed by the o11y team a while ago. This post documents the steps I took to create the karma rock.

Review the dockerfile

• It is a multi-build dockerfile, using nodejs and golang builders.
• With rockcraft we have parts and plugins.
• The analogue of COPY dockerfile directive is the prime directive.

First, let’s make sure we can manually build karma by following the dockerfile.

git clone --depth=1 https://github.com/prymitive/karma.git
cd karma

sudo snap install node --classic
cd ui
npm ci
touch node_modules/.install
make build
cd ..


sudo snap install go --classic
make download-deps-go
GO_ENABLED=0 make VERSION="0.120" karma

./karma --version

Create the rockcraft.yaml

Figure out the plugins available to use

At the moment of writing this, rockcraft doesn’t have many plugins documented, but from looking at snapcraft, it seems like the relevant plugins for us might be:

• dump
• make
• npm
• go

Start with a template

At this point it’s not exactly clear how the rock is going to turn out, but we do know already that:

1. We could have two parts, one with a node plugin and one with a go plugin. Let’s call them ui and cmd to match the dockerfile.
2. In the ui part we’d need to prime some files, and the cmd part will need those to build the executable.
3. In the end, we only stage the karma binary.
4. We need to declare a pebble service for karma. Let’s assume for now the binary is going to end up in /bin.
5. In CI, we’d need to automatically populate:

• source-tag, with the version tag we’re building from.
• build-snaps, with a go version compatible with what’s in karma’s go.mod file.

With this information we can start with a success-oriented rockcraft.yaml file:

name: karma
summary: prymitive karma in a ROCK.
description: Karma is an alert dashboard for Prometheus Alertmanager
version: "0.120"
base: ubuntu@22.04
license: Apache-2.0

services:
  karma:
    command: /bin/karma
    override: replace
    startup: enabled
platforms:
  amd64:
parts:
  ui:
    plugin: npm
    source: https://github.com/prymitive/karma.git
    source-type: git
    source-tag: "v0.120"
    source-depth: 1
    build-packages:
      - make
    override-build: |
      cd ui
      npm ci
      touch node_modules/.install
      make build
  cmd:
    after: [ ui ]
    plugin: go
    source: https://github.com/prymitive/karma.git
    source-type: git
    source-tag: "v0.120"
    source-depth: 1
    build-snaps:
      - go/1.22/stable
    override-build: |
      make download-deps-go
      GO_ENABLED=0 make VERSION="0.120" karma

Try to pack it

$ rockcraft pack --verbose

fails with the following message:

Environment validation failed for part 'ui': 'node' not found and part 'ui' does not depend on a part named 'npm-deps' that would satisfy the dependency.

Searching github for path:snapcraft.yaml content:npm-deps isn’t fruitful, so let’s take a look at the official docs (snapcraft):

• npm-include-node (bool, default: false) If true, download and include the node binary and its dependencies. If npm-include-node is true, then npm-node-version must be defined.
• npm-node-version (string) The version of node.js you want the snap to run on and includes npm, as would be downloaded from (https://nodejs.org). If~ not set, node.js must be provided another way, such as creating a node-deps part that provides node using build and staging dependencies or manually. Search GitHub for examples.

Let’s try using the same node version that comes from the currently stable snap:

parts:
  ui:
+   npm-include-node: true
+   npm-node-version: "20.12.2"

and try again:

$ rockcraft pack --verbose

This fails with:

:: /bin/bash: line 34: npm: command not found
'override-build' in part 'ui' failed with code 127.

Ok, npm should have been there. Let’s debug to see what’s wrong:

$ rockcraft pack --debug

rockcraft-karma-on-amd64-for-amd64-544665 ../project# find / -type f -name "node"

Nothing. Perhaps the plugin is not yet supported? Let’s manually add the snap:

parts:
  ui:
+   build-snaps:
+     - node/20/stable

Let’s try again:

$ rockcraft pack --verbose
...
Exported to OCI archive 'karma_0.120_amd64.rock'
Packed karma_0.120_amd64.rock

What’s inside the rock?

for f in $(tar tf karma_0.120_amd64.rock --wildcards 'blobs/sha256/*'); do tar -xvf karma_0.120_amd64.rock $f -O | tar tzf - >> files.txt; done

The karma pebble layer is there, but not karma itself. This makes sense because we didn’t tell rockcraft to keep it. Let’s try:

parts:
  cmd:
+   stage:
+     - karma

Packing fails with:

Failed to copy '/root/parts/cmd/install/karma': no such file or directory.

Let’s debug:

$ rockcraft pack --verbose --debug

rockcraft-karma-on-amd64-for-amd64-544665 ../project# ../parts/cmd/build/karma --version
0.120

Ok great, karma binary is built, we just need to figure out how to stage it correctly. So we know that karma is inside parts/cmd/build but is missing from install, which is why staging fails. Let’s manually copy it into install. From the craft-parts doc it seems like we need to:

    override-build: |
      make download-deps-go
      GO_ENABLED=0 make VERSION="0.120" karma
+     cp karma ${CRAFT_PART_INSTALL}

Now packing succeeds but now the karma binary is at the root filesystem of the oci archive. Let’s try to organize:

parts:
  cmd:
    stage:
      - karma
+   organize:
+     karma: bin/karma

Packing fails with:

Failed to copy '/root/parts/cmd/install/karma': no such file or directory.

Re-running with --debug, it turns out that now the karma binary is inside parts/cmd/install/bin. So seems like organize operates on the build root? And that we don’t need organize here at all?

Let’s take a look at a reference rock. Ah ha:

install -D -m755 bin/alertmanager ${CRAFT_PART_INSTALL}/bin/alertmanager

So instead of organize what we need here is:

• Arrange the $CRAFT_PART_INSTALL folder to our liking, and that’s how it will show up in the oci archive root.
• Use install as a shorthand for mkdir -p + cp + chmod.

parts:
  cmd:
    override-build: |
      make download-deps-go
      GO_ENABLED=0 make VERSION="0.120" karma
-     cp karma $CRAFT_PART_INSTALL
+     install -D -m755 karma $CRAFT_PART_INSTALL/bin/karma
    stage:
-     - karma
+     - bin/karma
-   organize:
-     karma: bin/karma

Packing succeeds and in the oci archive, the built & staged binary end up at /usr/bin/karma. Not sure what happens under the hood of rockcraft, but this is related to FHS.

Test the image

From the dockerfile we know we want port 8080, so:

sudo snap install docker
sudo skopeo --insecure-policy copy oci-archive:karma_0.120_amd64.rock docker-daemon:karma:0.120
sudo docker run --rm -d -p 8080:8080 karma:0.120
curl localhost:8080
# <!DOCTYPE html>
# (and so it goes...)

Success!

And the /usr symlink is indeed there:

$ sudo docker exec -it $(sudo docker run --rm -d -p 8080:8080 karma:0.120) /bin/bash
root@c287a93c6e1d:/# ls -l /
total 52
lrwxrwxrwx   1 root root    7 Apr 16 02:02 bin -> usr/bin
...

Note that the entry point to the container is pebble:

$ sudo docker run --rm -it karma:0.120
2024-04-19T03:06:57.711Z [pebble] Started daemon.
2024-04-19T03:06:57.721Z [pebble] POST /v1/services 4.804394ms 202
2024-04-19T03:06:57.727Z [pebble] Service "karma" starting: /bin/karma
2024-04-19T03:06:57.732Z [karma] level=info msg="Version: 0.120"

$ sudo docker run --rm -it karma:0.120 plan
2024-04-19T03:08:31.989Z [pebble] Started daemon.
2024-04-19T03:08:32.002Z [pebble] GET /v1/plan?format=yaml 105.005µs 200
services:
    karma:
        startup: enabled
        override: replace
        command: /bin/karma

Review the rockcraft.yaml file

Do we need the ui part at all?

It’s odd the we didn’t need to explicitly transfer anything from the ui part to the cmd part. Let’s try removing the entire ui part and see if it still builds (after rockcraft clean, just to be sure). Nope, it fails:

:: + GO_ENABLED=0
:: + make VERSION=0.120 karma
:: make[1]: Entering directory '/root/parts/cmd/build/ui'
:: make[1]: npm: No such file or directory

Let’s add the node snap and retry:

parts:
  cmd:
    build-snaps:
      - go/1.22/stable
+     - node/20/stable

Yep, that works!

Interim conclusions:

• make karma, which we use in the cmd part, is automatically building the ui too if it is not already built.
• For good measure, we’ll keep the ui part (npm) and the cmd part (go) separated.
• As of writing this, I do not know how the artifacts of the ui part are persisting for the cmd part without us explicitly specifying it.

Add boilerplate parts

In addition to the application itself, we also need:

• root CA certs installed to avoid TLS trust errors
• security manifest (temporary workaround)

parts:
  # ...
  
  ca-certs:
    plugin: nil
    overlay-packages: [ca-certificates]
  # The security manifest is required when .deb packages are added to the ROCK
  deb-security-manifest:
    plugin: nil
    after:
      - cmd
      - ca-certs
    override-prime: |
      set -x
      mkdir -p $CRAFT_PRIME/usr/share/rocks/
      (echo "# os-release" && cat /etc/os-release && echo "# dpkg-query" && dpkg-query --admindir=$CRAFT_PRIME/var/lib/dpkg/ -f '${db:Status-Abbrev},${binary:Package},${Version},${source:Package},${Source:Version}\n' -W) > $CRAFT_PRIME/usr/share/rocks/dpkg.query

And after that the oci archive indeed has etc/ssl/certs/* files.

Add CI

See observability workflows for ideas. For karma we need to make sure:

• go version matches go.mod
• node version matches packages.json
• version and source-tag are all consistent and get updated everytime the upstream project makes a release.
• oci-factory integration.

Summary of my knowledge gaps (loose ends) as of writing this

1. Do all “core” plugins (is there such a thing?), such as go, npm, etc., come from craft-parts and have the same UI across all starcraft projects?
2. What added value does a plugin bring if we use override-build?
3. How is npm-deps related (rockcraft mentions it in an error message)? It seems unused on github, and docs talk about npm-include-node, npm-node-version. Perhaps npm-deps is deprecated and the error string is outdated?
4. Even with npm-include-node, npm-node-version specified, node and npm were not installed. Does it mean the plugin isn’t implemented in rockcraft?
5. Should we be staging bin/* at all or just put files directly in usr/bin/* ourselves? Does it matter?
6. What magic enabled the artifacts of the ui part to remain available for the cmd part? I thought all parts are isolated unless explicitly transferred from one to another using the overlay directive.