Cross-controller end-to-end testing

charm
, , ,
1

Recently we integrated COS Lite (k8s bundle) with Data Platform’s machine charms. This means that integration / end-to-end tests would now need to be cross-controller tests.

In a recent PR we address this, with the goal to be able to run the e2e test both locally and in CI.

CI

actions-operator can be called repeatedly to create multiple controllers.

Each controller will be named after the provider used to bootstrap it (e.g. github-pr-ad8d9-microk8s). This means that we can currently bootstrap only one controller of each kind.

pytest-operator will pick up the last controller bootstrapped. To switch between controllers, we could save the CONTROLLER_NAME envvar that is set by the actions-operator action after each bootstrap.

  multi-controller-tests:
    name: microk8s-and-lxd-test
    runs-on: ubuntu-latest
    steps:
      - name: Setup k8s controller
        uses: charmed-kubernetes/actions-operator@main
        with:
          juju-channel: 3.0/stable
          provider: microk8s
          channel: 1.26-strict/stable

      - name: Save k8s controller name
        id: k8s-controller
        # The `CONTROLLER_NAME` envvar is set by this actions
        run: echo "name=$CONTROLLER_NAME" >> $GITHUB_OUTPUT

      - name: Setup lxd controller
        uses: charmed-kubernetes/actions-operator@main
        with:
          juju-channel: 3.0/stable
          provider: lxd

      - name: Save lxd controller name
        id: lxd-controller
        # The `CONTROLLER_NAME` envvar is set by this action
        run: echo "name=$CONTROLLER_NAME" >> $GITHUB_OUTPUT

      - name: Run integration tests
        run: ...
        env:
          K8S_CONTROLLER: ${{ steps.k8s-controller.outputs.name }}
          LXD_CONTROLLER: ${{ steps.lxd-controller.outputs.name }}

Note: there is a known issue deploying lxd next to microk8s on Juju 2.9 (this works with Juju 3).

Tox

For the tests to be able to make use of the different controllers, we need to tell tox to pass on the envvars:

[testenv:e2e]
description = Run end-to-end tests
passenv =
    K8S_CONTROLLER
    LXD_CONTROLLER

When running the test locally, we would need to bootstrap the controllers in advance, and then:

K8S_CONTROLLER="uk8s" LXD_CONTROLLER="lxd" tox -e e2e

Test code

Since we have two different models at play, we can’t rely on ops_test.model, because the auto-created model is created in the current controller, and we cannot know in advance which one it is going to be.

Instead, we would need to instantiate a Controller using the envvar. For example:

lxd_ctl_name = os.environ["LXD_CONTROLLER"]
lxd_ctl = Controller()
await lxd_ctl.connect(lxd_ctl_name)

Next, we need the model. pytest-operator has already created a random model name for us, but we do not know in which controller. So we use a helper function:

# The current model name is generated by pytest-operator from the test name + random suffix.  
# Use the same model name in both controllers.  
k8s_mdl_name = lxd_mdl_name = ops_test.model_name
k8s_mdl = await get_or_add_model(ops_test, k8s_ctl, k8s_mdl_name)
lxd_mdl = await get_or_add_model(ops_test, lxd_ctl, lxd_mdl_name)


async def get_or_add_model(ops_test: OpsTest, controller: Controller, model_name: str) -> Model:
    if model_name not in await controller.get_models():
        await controller.add_model(model_name)
        ctl_name = controller.controller_name
        await ops_test.track_model(
            f"{ctl_name}-{model_name}", cloud_name=ctl_name, model_name=model_name, keep=False
        )

    return await controller.get_model(model_name)

Establish cross-model relations

Before we can use “wait for idle”, we must relate any subordinate charms, because otherwise the subordiante would be in ‘unknown’ status. For example:

await asyncio.gather(
    lxd_mdl.add_relation("principal:juju-info", "subordinate"),
    lxd_mdl.block_until(lambda: len(lxd_mdl.applications["subordiante"].units) > 0),
)

After we deploy apps in both controllers, it is time to create offers and consume them:

await k8s_mdl.create_offer("grafana:dashbaords")
await lxd_mdl.consume(
    f"admin/{k8s_mdl.name}.grafana",
    application_alias="grafana",
    controller_name=k8s_ctl.controller_name,  # same as os.environ["K8S_CONTROLLER"]
)
await lxd_mdl.add_relation("agent", "grafana")

See full example of a cross-controller test here.

2 Likes