Community workshop: Fast & reusable GitHub CI

announcements and community
, ,
1

Recording: https://www.youtube.com/watch?v=iMHSbQfRw4k

What?

  • Faster integration tests by caching charm build & running tests in parallel
  • Strategies for sharing CI workflows across repositories (versioning, easy-to-maintain Python scripts)

When?

Friday 24 March 12:00–13:00 UTC

Where?

Video call link: https://meet.google.com/gzc-tuzr-ouv

Or dial: ‪(NL) +31 20 257 4201‬ PIN: ‪382 042 102‬#

More phone numbers: https://tel.meet/gzc-tuzr-ouv?pin=7066837377391

Further reading

Faster integration tests with GitHub caching

Faster CI results by running integration tests in parallel

Shared workflows: https://github.com/canonical/data-platform-workflows

5 Likes
2

I will be there!

3

Recording now available: https://www.youtube.com/watch?v=iMHSbQfRw4k

1 Like
4

I have tried out (with the help of) the @carlcsaposs reusable workflows today for the Nextcloud charm.

Massive thanx. I hope we can help improve this over time to work also on repos with more than one charm.

:pray:

5

@carlcsaposs

We had a talk in our team today and made some decisions based on the input I got from you yesterday during our call.

  1. We have to move away from having multiple charms in the same repo. While this is OK with us, it would be good to mention this in the docs that using this CI will not be ideal for repos with more than one charm.
  2. It would be good to mention that your repos have to be public for this to work. It would be very good if we could have private repos or possiby have some pipeline that does all work except publishing to charmhub.
  3. It would be good to mention how to retrieve the CHARMHUB_TOKEN and also mention the security implications for this. Also mention that it can be done on organizational level.
  4. How does this reusable CI work for “charm libraries”? Is there some release-yaml that takes care of publishing libraries?
  5. Its not 100% yet how to work with charms owned by a Launchpad team. I’m about to register a charm and need it to be possible to maintain by members of that team. How do you guys manage this in terms of CHARMHUB_TOKEN:s etc.

It will be great starting to add this now to our charms and get them out there. I hope to use this thread for reference.

@joakimnyman @jamesbeedy @marcus @tmihoc

6

When creating data-platform-workflows for our team’s repositories, we made a few decisions to simplify the workflows that limit the reusability for some configurations (e.g. repos with more than one charm, private repos). As we look to share these workflows more broadly with the charming community, it’s worth revisiting these limitations (and also improving the documentation). At the same time, supporting more configurations/addressing these limitations will require significant development work and I’m not sure how many resources we have for that work.

  1. This might change in the future, as our team has discussed the possibility of having some repos with more than one charm—but no decisions have been made about that yet.

To my knowledge, the only workflow that doesn’t work well for repos with more than one charm is release_charm.yaml, when called with create-github-release: true (the default). However, the workflows are currently only used/tested on repos with one charm.

  1. Unfortunately, this is mostly a limitation of GitHub: https://github.com/actions/toolkit/issues/1264. We made a decision to move a lot of the workflow logic to Python scripts to improve maintainability & reusability between workflows. However, this requires us to know the version of the workflow that’s running.

We found two possible solutions: (a) checking out the public caller repo or (b) specifying the workflow version twice when calling it—(b) would work for private repos, but it would cause issues if it gets out of sync with the other version. It’s possible to fork data-platform-workflows, and replace _get_workflow_version.yaml with a workflow input with the version. We might consider taking this approach in the future (or having it as a configurable option), but it would require some development work to create & maintain.

It may be possible to workaround this with the current workflows by setting the GITHUB_TOKEN to a personal access token or app token that has access to your repositories, but I’m not sure.

  1. This might be helpful: https://github.com/canonical/charming-actions/blob/main/release-libraries/README.md (this is a javascript github action, different from the python/yaml github workflows in data-platform-workflows)

  2. I have no experience with this—perhaps someone with more experience with Launchpad might be able to help.

Hope this helps give you some context & provide some possible options for ways to workaround the limitations we currently have.

Thank you for trying out the reusable workflows! The issues you’re running into with the workflows now are likely issues we’ll run into in the future—and learning/improving the workflows from your use case is also valuable to us.

1 Like