Kick-off integration tests with literal bundles

charm
1

Often, integration tests start with a cascade of deploy, wait-for-idle, add-relation, and wait-for-idle again. For example, in one of traefik’s itests we have:

async def test_build_and_deploy(ops_test: OpsTest, traefik_charm):
    await asyncio.gather(
        ops_test.model.deploy(
            traefik_charm,
            resources={
                "traefik-image": METADATA["resources"]["traefik-image"]["upstream-source"]
            },
            application_name="traefik"),
        ops_test.model.deploy(
            "ch:prometheus-k8s",
            application_name="prometheus",
            channel="edge",
            trust=True,
        ),
        ops_test.model.deploy(
            "ch:alertmanager-k8s",
            application_name="alertmanager",
            channel="edge",
            trust=True,
        ),
        ops_test.model.deploy(
            "ch:grafana-k8s",
            application_name="grafana",
            channel="edge",
            trust=True,
        ),
    )

    await ops_test.model.wait_for_idle(
        status="active", timeout=600, idle_period=30, raise_on_error=False
    )

    await asyncio.gather(
        ops_test.model.add_relation("prometheus:ingress", "traefik"),
        ops_test.model.add_relation("alertmanager:ingress", "traefik"),
        ops_test.model.add_relation("grafana:ingress", "traefik"),
    )

    await ops_test.model.wait_for_idle(status="active", timeout=600, idle_period=30)

We could potentially increase the congnitive ease by using a literal bundle:

async def test_build_and_deploy(ops_test: OpsTest, traefik_charm):
    test_bundle = dedent(f"""
        ---
        bundle: kubernetes
        name: test-tls
        applications:
          traefik:
            charm: {traefik_charm}
            trust: true
            resources:
              traefik-image: {METADATA["resources"]["traefik-image"]["upstream-source"]}
          prometheus:
            charm: prometheus-k8s
            trust: true
            channel: edge
          alertmanager:
            charm: alertmanager-k8s
            trust: true
            channel: edge
          grafana:
            charm: grafana-k8s
            trust: true
            channel: edge

        relations:
        - [traefik:ingress, alertmanager:ingress]
        - [traefik:ingress-per-unit, prometheus:ingress]
        - [traefik:traefik-route, grafana:ingress]
    """)
    await deploy_literal_bundle(ops_test, test_bundle)  # See appendix below
    await ops_test.model.wait_for_idle(status="active", timeout=600, idle_period=30)

Pros

  • All the information is captured in a standardized format – the bundle.
  • Reduce the amount of the gather/await needed.
  • No need to concern ourselves with code ordering of deploy and add_relation.
  • Potentially better consistency (and less duplication) across multiple tests.

Cons

  • Similar line count for both cases, but the literal yaml isn’t as easily programmable.
  • Validating/linting the literal yaml isn’t starightforward, unlike regular code, for which we could have static checks.

Appendix

The deploy_literal_bundle function is a workaround for ops_test.deploy_bundle:

async def deploy_literal_bundle(ops_test: OpsTest, bundle: str):
    run_args = [
        "juju",
        "deploy",
        "--trust",
        "-m",
        ops_test.model_name,
        str(ops_test.render_bundle(bundle)),
    ]

    retcode, stdout, stderr = await ops_test.run(*run_args)
    assert retcode == 0, f"Deploy failed: {(stderr or stdout).strip()}"
    logger.info(stdout)
1 Like
2

I was thinking yesterday about something very similar. What I had in mind was to write a pytest plugin that would be able to ingest something like this:

folder structure:

tests
  - integration
    - test-1
      - bundle.yaml
      - test_foo.py
      - test_bar.py
    - test-2
      - bundle.yaml
      - test_baz.py

The plugin would: for each subfolder in integration/: deploy the bundle and run the tests with that bundle. Pros:

  • can parallelize across bundles

Cons:

  • many of our tests dynamically deploy/remove an application and verify that the right stuff happens, that we’d still need to do manually
3

Nice idea! Q: can the specific revision/series by deployed this way?

P.S. We are affected a lot by this issue.

4

Hi @taurus,

IIRC, @robgibbon may have more info about charm revision pinning.

5

can the specific revision/series by deployed this way?

From mysql-bundle.yaml I gather that revision pinning is supported. So yes, the example here is a regular bundle and you can have there anything that bundles support.

6

for each subfolder in integration/: deploy the bundle and run the tests with that bundle

That could be very neat. Looks like:

  • We can parallelize with xdist.
  • To have them spread across several VMs in CI we would still need something like DPE’s CI.
1 Like
7

If I’m not mistaken, pytest-operator provides utilities for deploying Juju bundles, specifically deploy_bundle(...). Is there a reason you wouldn’t be able to use this feature?

The current problem with pytest operator’s deploy_bundle(...) is that it relies on an outdated Rust binary called juju-bundle to deploy the bundle. IIRC juju-bundle was the predecessor to bundle support in Juju. I actually fixed this issue in https://github.com/charmed-kubernetes/pytest-operator/pull/99, however, the pull request is still a work-in-progress because my changes are rather invasive and require releasing a major revision of pytest-operator.

Let me know if my prior work on pytest-operator could potentially help with your integration tests. If so, I will try to make time to finish the changes to pytest-operator.

1 Like
8

Correct, deploy_bundle depends on juju-bundle, which you have to know to install separately.

I’m glad to hear there’s an upcoming fix!