Charming DEBUG on GitHub

, ,
1

Hi,

This is a short article about our charm debugging experience on GitHub (to share with the team). Many tests are failing on GitHub for "unknown reason"© producing some traceback and forcing developers to invest hours in reproducing the issue locally (usually without luck).

Be aware of your possibilities:

  1. you can enable debug RUNNERS and STEPS on GitHub using repo secret ACTIONS_RUNNER_DEBUG=true and ACTIONS_STEP_DEBUG=true.

  2. you should enable units DEBUG for the test model logging-config for your tests, e.g.:

      - name: Setup operator environment
        uses: charmed-kubernetes/actions-operator@main
        with:
          provider: microk8s
          bootstrap-options: "--model-default 'logging-config=<root>=INFO;unit=DEBUG' --agent-version 2.9.42"

Another option is to set it inside the unit test:

  MODEL_CONFIG = {"logging-config": "<root>=INFO;unit=DEBUG"}
  await ops_test.model.set_config(MODEL_CONFIG)
  1. you can SSH into the GitHub runner and trace the failure in place with “lhotari/action-upterm”:
      - name: Setup upterm session
        if: success() || failure()
        uses: lhotari/action-upterm@v1
        with:
          limit-access-to-actor: true
          limit-access-to-users: your-github-nick

It will give you SSH URI to connect:

SSH Session: ssh vCwVzfRGKw6gCZ9r2S9e:MWNlNmQ0OGEudm0udXB0ZXJtLmludGVybmFsOjIyMjI=@uptermd.upterm.dev

Simply SSH there using your GitHub SSH key and continue juju/charm/microk8s troubleshooting commonly.

HINTS:

  • make sure you are testing with “–keep-models” option to keep the model running on tests failure/successes!

  • make sure you set DEBUG level for charm: logging-config=<root>=INFO;unit=DEBUG (before running the test(s)).

    NOTE: GH runner will be cleaned on SSH logout and on timeout (6 hours by default).

  1. you can fetch logs locally using CLI tool: gh run view RUN_ID --log (e.g. to upload in a ticket the entire log). Replace RUN_ID with the ID from view logs URL, e.g.: ~https://github.com/canonical/your-repo/actions/runs/4548879168/jobs/8030561836

  2. be aware of the default charmed-kubernetes/actions-operator behavior --model-default automatically-retry-hooks=false. Ops will abort testing on the first (and possibly temporary) charm state “error”. While locally, it is ignored by default due to automatically-retry-hooks=true.

Have fun with charming debug on GitHub!!!

3 Likes
2

upterm is excellent! Just don’t forget to touch /continue once you’ve started the session to move to the next step.

I will add, at least for me it only worked while in an active tmux session, with PasswordAuthenticaion yes and PubkeyAuthentication yes in my /etc/ssh/sshd_config. Might be OS specific.

1 Like