smoke testing with Scenario

charm
, ,
1

A little-known experimental feature in scenario is check_builtin_sequences. That is a function that essentially fires every possible event on your charm and hopes for the best.

Of course, that the charm does not raise any exception is a very low bar to set, but an important one nonetheless. It could help detect simple bugs and it’s a relatively inexpensive test to run, so, why not?

from scenario import State
from scenario.sequences import check_builtin_sequences

def test_builtin_events(mycharm):
    ctx, _ = check_builtin_sequences(
        mycharm,
        meta={"name": "foo",
              "requires": {"foo": {"interface": "bar"}},
              "containers": {"baz": {"image": "tuktuk"}}
              },
        config={"options": {"foo": {"type": "string"}}},
        template_state=State(
            config={"foo": "bar"},
            relations=[
                Relation('foo'),
            ],
            containers=[
                Container('baz')
            ]),
    )
    assert len(ctx.emitted_events) == 36

This rather simple configuration already tests your charm with 36 different events! There’s very little generic assertions worth making for every possible events in the general case, I suppose, except perhaps asserting that the charm set some sort of non-Unknown status.

What do you think?

3 Likes
2

Would it make sense to provide the 36 magic number in scenario as well, so that if events are added or removed, it does not break the tests?

3

Would be nice to eventually have something like this in the charmcraft init template.

4

unfortunately the number of events fired depends on the input state. For each container you have, for example, you get a pebble-ready. For each relation, you get (-changed, -departed, -joined…) so that’s 5.

I’m not suggesting by the way that asserting ‘this many events have been fired’ is a meaningful test. It was simply a way to show how many events were fired in this specific case. As I mention in the post, there’s very little you can generically assert about all of those events. The most important part of this kind of testing is verifying that the charm isn’t raising uncaught exceptions.

5

yeah that’s a nice idea. So far the template contains tests that you have to delete anyway. Would be nice to have some tests you can actually keep.