Charmcraft's upload-resource command

[facundo (Facundo Batista)]

This post is about the desired behaviour for the upload-resource command in Charmcraft. Some of the functionality is there, other needs to be implemented.

Related issues: , #263, #350

Spec

The upload-resource command will support both resource types, file and oci-image.

The signature of the command will be:

charmcraft upload-resource charm-name resource-name [--filepath=PATH|--image=DIGEST]

The charm and resource names identifies the resource for which charmcraft will upload the content, and are both mandatory parameters. After that, one of --filepath or --image must be specified, for file and oci-image resource types correspondingly.

In both cases the developer needs to own or be a collaborator of the charm related to the resource that is uploading.

For the file case

The option --filepath must indicate a readable local file; charmcraft will just upload that content to Charmhub.

For the OCI image case

The option --image must indicate an OCI image’s digest, being it in the short or long form (e.g.: 70aa8983ec5c or sha256:70aa8983ec5cea7bc143af188b4bdc424fa405184d56dcbdfd9df672ade85249). Note that the image name is not needed (as the resource is already identified) and a tag is not allowed (the image needs to be univocally identified). Likewise, only a digest is accepted, not an URL to generic external registry.

For the case of the OCI image, charmcraft will first check if that specific image is available in the Canonical’s Registry, and just use it if that’s the case. If not, it will try to get it from the local OCI repository, push it to the Canonical’s Registry, and then use it. Otherwise it will fail with a proper error message. Note that in the case of using the “short form” of the digest, the image needs to be present locally so its proper ID (the “long form”) could be retrieved.

The following is the case of charmcraft using the image already present in the Canonical’s registry:

$ charmcraft upload-resource my-charm my-resource --image=sha256:c8f0dbc0d5a5e3aef7281a4d3941e97f204390b2cba9de6c117341361cbe84e4
Using the OCI image from Canonical's registry
Revision 7 created of resource 'my-resource' for charm 'my-charm'

The following is the case of charmcraft needing to push the image first:

$ charmcraft upload-resource mycharm my-resource --image=sha256:70aa8983ec5cea7bc143af188b4bdc424fa405184d56dcbdfd9df672ade85249
Remote image not found, uploading from local registry
Image uploaded, new remote digest: sha256:29b027d656acae6e50693a2ec063d40c7de59a2f7b4c0d62c765d2f1cbde616c
Revision 5 created of resource 'my-resource' for charm 'my-charm'

One detail to take into consideration. When charmcraft pushes the local image to Canonical’s registry, the remote manifest ends up slightly different to the local one. This is deterministic, so future operations will continue to use the same manifest, but it does take some work each time to compute the target manifest.

A note about authentication

As said above in the general case, the developer needs to own or be a collaborator of the related charm.

This is specially true when working with the Canonical’s registry. All the interactions with the registry are authenticated, and the images there live under a path that includes the charm id.

So when charmcraft goes to the registry and asks for a digest (to reuse it), it would really check if there’s an image for that charm (that the client can see) with that digest. Similarly, when an image is uploaded, it will be uploaded only for the indicated charm.

[jameinel (John A Meinel)]

Thanks for documenting upload-resource.

Just to confirm, charmcraft upload-resource --image will only support uploading a hash identified in a local repository, not one that is specified by a remote URL, is that correct?

[facundo (Facundo Batista)]

Yes! I just added that clarification to the text.

[niemeyer (Gustavo Niemeyer)]

Thanks for the details, Facundo.

@facundo

charmcraft upload-resource charm-name resource-name

Do we always need the charm name? Aren’t other commands assuming the charm lives in the current working directory? If so, it might be a good idea to make it an optional non-positional argument that defaults to that same behavior.

@facundo

For the case of the OCI image, charmcraft will first check if that specific image is available in the Canonical’s Registry, and just use it if that’s the case. If not, it will try to get it from the local OCI repository

How do we handle privacy issues for images the client doesn’t actually hold? That is, knowledge of a digest doesn’t imply possession of the bits. To be able to reuse content like this something must ensure that either the bits are necessarily public content anyway, or that the client indeed has the bits at hand. Alternatively, a cheap workaround in terms of implementation is to just push the bits again for now.

Whatever the case, it would be good to have some words in the spec about such details.

[facundo (Facundo Batista)]

@niemeyer

Do we always need the charm name? Aren’t other commands assuming the charm lives in the current working directory? If so, it might be a good idea to make it an optional non-positional argument that defaults to that same behavior.

Yes, we need the charm name. All store-related commands that needs the charm name have been migrating to always include it because they get simpler and more consistent (and more aligned with other tools, e.g. snapcraft)

@niemeyer

How do we handle privacy issues for images the client doesn’t actually hold? That is, knowledge of a digest doesn’t imply possession of the bits. To be able to reuse content like this something must ensure that either the bits are necessarily public content anyway, or that the client indeed has the bits at hand. Alternatively, a cheap workaround in terms of implementation is to just push the bits again for now.

All interactions with the registry are authenticated, and the images in Canonical’s registry live under a path that includes the charm id. So when charmcraft goes to the registry and asks for a digest, it would really check if there’s an image for that charm (that the client can see) with that digest.

For example, if I run this command:

charmcraft -v upload-resource facundo-snappass-test redis-image --image=sha256:690e4e55731042bf5f8dfba2527b0459ec4f298f9582fd466718262f864e403f

It will authenticate using this scope: repository:charm/1ghikkzuhwzekkxgjhj2n5y5v07mqwhn805ym/redis-image:pull

And then check for the image doing a POST to https://api.staging.charmhub.io/v1/charm/facundo-snappass-test/resources/redis-image/oci-image/blob with body {'image-digest': 'sha256:690e4e55731042bf5f8dfba2527b0459ec4f298f9582fd466718262f864e403f'}

I’ll add a clarification in the Spec about “the developer needing to own or be a collaborator of the charm that wants to upload a image to”.

Thanks!

[niemeyer (Gustavo Niemeyer)]

@facundo

Yes, we need the charm name. All store-related commands that needs the charm name have been migrating to always include it because they get simpler and more consistent (and more aligned with other tools, e.g. snapcraft)

The most analogous commands I can think of, snapcraft upload and snapcraft upload-metadata, do not take a snap name and instead introspect the file. That’s in part what made me wonder. But I see the list-* commands do take a snap name. Either way, sounds fine.

@facundo

All interactions with the registry are authenticated, and the images in Canonical’s registry live under a path that includes the charm id. So when charmcraft goes to the registry and asks for a digest, it would really check if there’s an image for that charm (that the client can see) with that digest.

Can we please have that in the spec and mention the security issue explicitly, so that conversation and intention is preserved? Right now it talks about the image existing in “Canonical’s registry”, which sounds broader than what’s being described now.

[facundo (Facundo Batista)]

@niemeyer

The most analogous commands I can think of, snapcraft upload and snapcraft upload-metadata, do not take a snap name and instead introspect the file. That’s in part what made me wonder. But I see the list-* commands do take a snap name. Either way, sounds fine.

It’s similar to what happens in charmcraft upload: as you have to specify the binary already, information can be extracted from there. But all the rest of the commands doesn’t have that binary, and may not even be run in the project’s directory.

@niemeyer

Can we please have that in the spec and mention the security issue explicitly, so that conversation and intention is preserved? Right now it talks about the image existing in “Canonical’s registry”, which sounds broader than what’s being described now.

I added this info to the spec (see the “A note about authentication” last point).

Thanks!

[jameinel (John A Meinel)]

@facundo

As said above in the general case, the developer needs to own or be a collaborator of the related charm.

the developer needs to be an owner or collaborator on the charm

?

[jameinel (John A Meinel)]

@facundo

One detail to take in consideration is that if charmcraft pushes the local image to the Canonical’s registry, as the remote manifest is different than the local one, the image’s digest will change. So, if the same exact command is repeated, it will do quite some work: open the local image, zip the layers, hash everything, etc; no upload would really happen, but the command will not be just a noop.

One detail to take into consideration. When charmcraft pushes the local image to Canonical’s registry, the remote manifest ends up slightly different to the local one. This is deterministic, so future operations will continue to use the same manifest, but it does take some work each time to compute the target manifest.

FWIW, I am also concerned that this is an impediment to users. They will have done testing an validation on a given hash. They can then record that “abcdef” is a known good situation, and then 'charmcraft upload-resource … sha256:abcdef". But then once they deploy that charm, they will see in ‘kubectl describe’ a different hash, and won’t have an easy way to correlate that back with their CI/CD system.

Is there any way that we can avoid having to modify the image as part of the upload?

If we cannot, then we need to be very clear with our messaging during ‘upload-resource’ so that they can have a clear handle of the final hash they will be running in production.

[facundo (Facundo Batista)]

Yes, they need those permissions. IOW, I will not be able to upload a resource to your charm (unless I’m a collaborator, that is).

[facundo (Facundo Batista)]

Thanks for the wording.

I think there’s no way for the image to be different, because even the layers being exactly the same, the manifest is changing, so the hash will be different…

Regarding the user knowing the new digest, see that the command’s output includes this new remote digest.

[guoqiao (Joe Guo)]

I followed the resource upload process, but when I run juju deploy $APP, image pulling failed with this message:

Failed to pull image 
"registry.jujucharms.com/charm/gvve6snw6vluyzxaxd1yqevxd1lj18nncr3de/pihole-image@sha256:a4ea08eb6fe4b4c12b23ba88752c36181e49387ec4620b88f67a09673038a8d0": 
rpc error: code = Unknown desc = failed to pull and unpack image "registry.jujucharms.com/charm/gvve6snw6vluyzxaxd1yqevxd1lj18nncr3de/pihole-image@sha256:a4ea08eb6fe4b4c12b23ba88752c36181e49387ec4620b88f67a09673038a8d0": 
failed to resolve reference "registry.jujucharms.com/charm/gvve6snw6vluyzxaxd1yqevxd1lj18nncr3de/pihole-image@sha256:a4ea08eb6fe4b4c12b23ba88752c36181e49387ec4620b88f67a09673038a8d0": 
pull access denied, repository does not exist or may require authorization: 
server message: insufficient_scope: authorization failed

So far, I am not sure the issue is caused by charmcraft, juju, charmhub, or local env.
Am I alone ?

[facundo (Facundo Batista)]

You’re not alone, it’s a bug in Juju, looks it will be solved in 2.9.8, track it here. Thanks!

[sed-i (Leon)]

@facundo

only a digest is accepted

Maybe it is worthwhile to mention that the argument to --image should be the output of:

docker inspect alpine --format '{{ .Id }}'

[facundo (Facundo Batista)]

@sed-i

–format ‘{{ .Id }}’

Note you can also just use the truncated hash from the IMAGE ID column when you do docker images.

[cbelu (Claudiu Belu)]

Hello,

I do have a concern when it comes to uploading resources, OCI images to more specific.

My expectation was that if an image was already uploaded to Charmhub, the next time I’d try to upload the next image it would basically be a noop operation that wouldn’t take more than a few seconds.

Yet, that is not what’s happening. As you can see in these 2 image jobs [1][2], the same images have been uploaded twice, which took some time. You can see that it is the same image being uploaded from the logs:

2022-03-18T10:00:08.0307023Z Status checked: {'revisions': [{'errors': None, 'revision': 5, 'status': 'appr…+ charmcraft upload-resource finos-legend-sdlc-k8s sdlc-image --image=sha256:9d417201eeab7a949556f7819162239c453048bd912a19d8c1bc15d604cc9afc
2022-03-18T10:00:10.6095048Z Uploading resource from image charm/1lzc39g08fl38dd2r6hpvaxyisc2prwtus1gc/sdlc…
2022-03-18T10:00:11.9032497Z 
2022-03-18T10:00:11.9037052Z Remote image not found, uploading from local registry.

and:

2022-03-18T10:58:21.7256045Z Status checked: {'revisions': [{'errors': None, 'revision': 6, 'status': 'appr…+ charmcraft upload-resource finos-legend-sdlc-k8s sdlc-image --image=sha256:9d417201eeab7a949556f7819162239c453048bd912a19d8c1bc15d604cc9afc
2022-03-18T10:58:23.6035574Z Uploading resource from image charm/1lzc39g08fl38dd2r6hpvaxyisc2prwtus1gc/sdlc…
2022-03-18T10:58:25.5829028Z 
2022-03-18T10:58:25.5831755Z Remote image not found, uploading from local registry.

Now, what’s interesting, is that eventually charmcraft / charmhub figures out that it already has these layers / images:

...
2022-03-18T10:02:05.8005812Z Extracting file '898f59840ea54d3f7e5e1a363041f067a04842a71f407139375a79260af4b…
2022-03-18T10:02:05.8006902Z Uploading layer blob 7/8, size=202188869, digest=sha256:7775937d0b283ccb76ffe4…
2022-03-18T10:02:08.0844183Z Checking if the blob is already uploaded                                       
2022-03-18T10:02:08.1172035Z Blob was already uploaded                                                      
2022-03-18T10:02:18.5387717Z Extracting file 'f3f13879c8412d0a02e576d17fcca360f7fc6c9fe342a9d7b487e6c7aca98…
2022-03-18T10:02:18.5388817Z Uploading layer blob 8/8, size=170912297, digest=sha256:e176b9f2ff9e6e83a069c8…
2022-03-18T10:02:19.4293069Z Checking if the blob is already uploaded                                       
2022-03-18T10:02:19.4618319Z Blob was already uploaded                                                      
2022-03-18T10:02:21.7777399Z Uploading manifest with reference sha256:304e7c5bec85a7f57adfeb8c366cd40e04469…
2022-03-18T10:02:21.9095000Z 
2022-03-18T10:02:23.6717418Z Image uploaded, new remote digest: sha256:304e7c5bec85a7f57adfeb8c366cd40e044698448554e31bbfddc73ed25eb6a2.

And yes, the digest sha256:304e7c5bec85a7f57adfeb8c366cd40e044698448554e31bbfddc73ed25eb6a2 is the same in both jobs.

Now, what happened here is that the digest in the charmhub registry sha256:304e7c5bec85a7f57adfeb8c366cd40e044698448554e31bbfddc73ed25eb6a2 is different from the image digest I’m uploading sha256:9d417201eeab7a949556f7819162239c453048bd912a19d8c1bc15d604cc9afc, which results in the image getting uploaded again. This will also result in having new resource revisions (even though it’s the same), which makes it harder to track / maintain.

Ideally, this wouldn’t have happened in the first place. It needlessly consumes CPU time and bandwidth for both us and the Charmhub servers, and makes it harder to track resource revisions.

Additionally, even more time / bandwidth would have been saved if we didn’t have to pull the images locally before pushing them to the Charmhub registry, if there was an option to just have Charmhub mirror those images directly.

[1] https://github.com/claudiubelu/legend/runs/5598551265?check_suite_focus=true [2] https://github.com/claudiubelu/legend/runs/5599231160?check_suite_focus=true

[facundo (Facundo Batista)]

Hello!

This situation occurs because the local image does not have the same manifest than the remote one. The local image is just a bunch of layers blobs, which needs to be uploaded (which is a noop in each case if the server already has it) and then the created manifest is also uploaded for the registry to work ok.

So, in reality, we don’t have nothing locally to compare to the server and decide that “exactly all this image/content is already there”.

The detail about not mirroring a external resource, so needing to always upload from local resources is more a legal issue than technical (the developer, authenticated as such, is uploading content from their machine).

Sorry for the inconvenience. Thanks!