Charmcraft's upload-resource command

This post is about the desired behaviour for the upload-resource command in Charmcraft. Some of the functionality is there, other needs to be implemented.

Related issues: , #263, #350

Spec

The upload-resource command will support both resource types, file and oci-image.

The signature of the command will be:

charmcraft upload-resource charm-name resource-name [--filepath=PATH|--image=DIGEST]

The charm and resource names identifies the resource for which charmcraft will upload the content, and are both mandatory parameters. After that, one of --filepath or --image must be specified, for file and oci-image resource types correspondingly.

In both cases the developer needs to own or be a collaborator of the charm related to the resource that is uploading.

For the file case

The option --filepath must indicate a readable local file; charmcraft will just upload that content to Charmhub.

For the OCI image case

The option --image must indicate an OCI image’s digest, being it in the short or long form (e.g.: 70aa8983ec5c or sha256:70aa8983ec5cea7bc143af188b4bdc424fa405184d56dcbdfd9df672ade85249). Note that the image name is not needed (as the resource is already identified) and a tag is not allowed (the image needs to be univocally identified). Likewise, only a digest is accepted, not an URL to generic external registry.

For the case of the OCI image, charmcraft will first check if that specific image is available in the Canonical’s Registry, and just use it if that’s the case. If not, it will try to get it from the local OCI repository, push it to the Canonical’s Registry, and then use it. Otherwise it will fail with a proper error message. Note that in the case of using the “short form” of the digest, the image needs to be present locally so its proper ID (the “long form”) could be retrieved.

The following is the case of charmcraft using the image already present in the Canonical’s registry:

$ charmcraft upload-resource my-charm my-resource --image=sha256:c8f0dbc0d5a5e3aef7281a4d3941e97f204390b2cba9de6c117341361cbe84e4
Using the OCI image from Canonical's registry
Revision 7 created of resource 'my-resource' for charm 'my-charm'

The following is the case of charmcraft needing to push the image first:

$ charmcraft upload-resource mycharm my-resource --image=sha256:70aa8983ec5cea7bc143af188b4bdc424fa405184d56dcbdfd9df672ade85249
Remote image not found, uploading from local registry
Image uploaded, new remote digest: sha256:29b027d656acae6e50693a2ec063d40c7de59a2f7b4c0d62c765d2f1cbde616c
Revision 5 created of resource 'my-resource' for charm 'my-charm'

One detail to take into consideration. When charmcraft pushes the local image to Canonical’s registry, the remote manifest ends up slightly different to the local one. This is deterministic, so future operations will continue to use the same manifest, but it does take some work each time to compute the target manifest.

A note about authentication

As said above in the general case, the developer needs to own or be a collaborator of the related charm.

This is specially true when working with the Canonical’s registry. All the interactions with the registry are authenticated, and the images there live under a path that includes the charm id.

So when charmcraft goes to the registry and asks for a digest (to reuse it), it would really check if there’s an image for that charm (that the client can see) with that digest. Similarly, when an image is uploaded, it will be uploaded only for the indicated charm.

2 Likes

Thanks for documenting upload-resource.

Just to confirm, charmcraft upload-resource --image will only support uploading a hash identified in a local repository, not one that is specified by a remote URL, is that correct?

Yes! I just added that clarification to the text.

Thanks for the details, Facundo.

Do we always need the charm name? Aren’t other commands assuming the charm lives in the current working directory? If so, it might be a good idea to make it an optional non-positional argument that defaults to that same behavior.

How do we handle privacy issues for images the client doesn’t actually hold? That is, knowledge of a digest doesn’t imply possession of the bits. To be able to reuse content like this something must ensure that either the bits are necessarily public content anyway, or that the client indeed has the bits at hand. Alternatively, a cheap workaround in terms of implementation is to just push the bits again for now.

Whatever the case, it would be good to have some words in the spec about such details.

Yes, we need the charm name. All store-related commands that needs the charm name have been migrating to always include it because they get simpler and more consistent (and more aligned with other tools, e.g. snapcraft)

All interactions with the registry are authenticated, and the images in Canonical’s registry live under a path that includes the charm id. So when charmcraft goes to the registry and asks for a digest, it would really check if there’s an image for that charm (that the client can see) with that digest.

For example, if I run this command:

charmcraft -v upload-resource facundo-snappass-test redis-image --image=sha256:690e4e55731042bf5f8dfba2527b0459ec4f298f9582fd466718262f864e403f

It will authenticate using this scope: repository:charm/1ghikkzuhwzekkxgjhj2n5y5v07mqwhn805ym/redis-image:pull

And then check for the image doing a POST to https://api.staging.charmhub.io/v1/charm/facundo-snappass-test/resources/redis-image/oci-image/blob with body {'image-digest': 'sha256:690e4e55731042bf5f8dfba2527b0459ec4f298f9582fd466718262f864e403f'}

I’ll add a clarification in the Spec about “the developer needing to own or be a collaborator of the charm that wants to upload a image to”.

Thanks!

The most analogous commands I can think of, snapcraft upload and snapcraft upload-metadata, do not take a snap name and instead introspect the file. That’s in part what made me wonder. But I see the list-* commands do take a snap name. Either way, sounds fine.

Can we please have that in the spec and mention the security issue explicitly, so that conversation and intention is preserved? Right now it talks about the image existing in “Canonical’s registry”, which sounds broader than what’s being described now.

It’s similar to what happens in charmcraft upload: as you have to specify the binary already, information can be extracted from there. But all the rest of the commands doesn’t have that binary, and may not even be run in the project’s directory.

I added this info to the spec (see the “A note about authentication” last point).

Thanks!

the developer needs to be an owner or collaborator on the charm

?

One detail to take into consideration. When charmcraft pushes the local image to Canonical’s registry, the remote manifest ends up slightly different to the local one. This is deterministic, so future operations will continue to use the same manifest, but it does take some work each time to compute the target manifest.

FWIW, I am also concerned that this is an impediment to users. They will have done testing an validation on a given hash. They can then record that “abcdef” is a known good situation, and then 'charmcraft upload-resource … sha256:abcdef". But then once they deploy that charm, they will see in ‘kubectl describe’ a different hash, and won’t have an easy way to correlate that back with their CI/CD system.

Is there any way that we can avoid having to modify the image as part of the upload?

If we cannot, then we need to be very clear with our messaging during ‘upload-resource’ so that they can have a clear handle of the final hash they will be running in production.

1 Like

Yes, they need those permissions. IOW, I will not be able to upload a resource to your charm (unless I’m a collaborator, that is).

Thanks for the wording.

I think there’s no way for the image to be different, because even the layers being exactly the same, the manifest is changing, so the hash will be different…

Regarding the user knowing the new digest, see that the command’s output includes this new remote digest.