Parameterized deployments in integration tests

charm
, , ,
1

In parallel to enabling TLS in COS Lite, we would like to have a convenient way to re-run existing integration test, but with TLS enabled. TLS is ubiquitous and most charm’s features should probably work just the same with or without TLS enabled.

So how can we repeat the same tests with and without TLS?

In a recent PR we propose the following.

  1. Use SimpleNamespace so strings are more “static”, for example:
    am = SimpleNamespace(name="am", charm="alertmanager-k8s", scale=1)
  2. Have a function that returns different literal bundles, for example:
    def bundle_under_test(charm_under_test, tls_enabled: bool) -> str.
  3. Parametrize the deployment test function with module scope, for example:
    @pytest.mark.parametrize("tls_enabled", [False, True], scope="module").
  4. Remove all apps at the end, so the model is clean before the next parameterization runs.

Note the scope="module" of the parametrize decorator:

    :arg scope: if specified it denotes the scope of the parameters.  
        The scope is used for grouping tests by parameter instances.

This way the parametrized items are the “outer loop”. Without scope="method", pytest would run all the parameterized items on the same test function and only then move forward to the next test method, which is not what we want.

Here’s how a test sequence could look like with module scope parameterization:

def bundle_under_test(charm_under_test, tls_enabled: bool) -> str:
    without_tls = ...
    with_tls = ...
    return with_tls if tls_enabled else without_tls


@pytest.mark.parametrize("tls_enabled", [False, True], scope="module")  
@pytest.mark.abort_on_fail  
async def test_build_and_deploy(ops_test, charm_under_test, tls_enabled):
    # `charm_under_test` is packed locally; the rest is from charmhub.
    await deploy_literal_bundle(
        ops_test, bundle_under_test(charm_under_test, tls_enabled)
    )
    ...


@pytest.mark.abort_on_fail  
async def test_remove(ops_test):
    ...

Similarly, we could group the tests in a class and parametrize with a “class” scope.

@pytest.mark.parametrize("tls_enabled", [False, True], scope="class")  
class TestCharm:  
  
    @pytest.mark.abort_on_fail  
    async def test_build_and_deploy(self, ops_test, charm_under_test, tls_enabled):
        await deploy_literal_bundle(ops_test, bundle_under_test(
            charm_under_test, tls_enabled)
        )
        ...
        
    @pytest.mark.abort_on_fail  
    async def test_remove(self, ops_test, tls_enabled):
        ...

You can compare these options with the PR’s commits:

  1. c935c51 Add parametrized TLS integration test
  2. b657798 Class-scope parameterization
  3. 3dfcc7b Module-scope parameterization