Classify tests with pytest custom markers for quick Integration testing iterations

I’ve done a poor man’s version of this for a while and its really helpful. I might do tox -e integration -- --model my-model once, and if it fails on a test I’d rerun it with tox -e integration -- -k "not build_and_deploy" or tox -e integration -- -k "test_that_failed". But using markers with a standard nomenclature is a much nicer way, especially when setup occurs across several tests. If I could find the time, I’d love to put these markers in all the Kubeflow charms

I’ve wanted to do this too for cases where a bundle can benefit from a charm’s tests. It would be nice for a bundle’s tests to be able to do:

  • deploy bundle
  • git clone charm-in-bundle
  • cd charm-in-bundle; tox -e integration -- -k "not setup" # ← test the bundle using a charm’s tests

That could help spot configuration errors that bundles introduce to break charms without literally duplicating tests unnecessarily