Can decision models make CI more efficient? Experimenting with Jev in Vault
Continuous integration (CI) is an essential part of software development, but as a project grows, so does the cost of maintaining and running its tests.
This is something we have been dealing with in the Canonical Vault charm repository. Vault has both Kubernetes and machine charms, shared code, tests for our monitoring dashboards, linters, and integration tests across multiple architectures.
The integration tests are particularly expensive because each test job provisions its own environment, involving components such as Juju, LXD, MicroK8s, and Vault.
Originally, we had a large integration test suite that took a long time to complete. This meant developers had to wait for the entire suite to finish before getting useful feedback. To address this, we split the tests into smaller jobs that could run in parallel.
While this improved feedback times, it introduced another problem: we now had many jobs provisioning their own environments, consuming a significant amount of runner time and computing resources.
We started exploring ways to make our CI more selective. More recently, we began experimenting with Jev, a decision model, to see whether it could help us reduce unnecessary CI execution and automatically recover from intermittent failures.
In this article, we’ll share the approach, the implementation, and some of the challenges we encountered. This is still an experiment, and we’ll evaluate its effectiveness after deploying it in Vault.
The limitations of rule-based CI optimization
Our first approach was to introduce rules that determine which workflows need to run based on the files changed in a pull request.
For example, if a change only affects the Kubernetes charm, there is no reason to run integration tests for the machine charm. Similarly, documentation-only changes should not require provisioning an integration environment.
These optimizations helped reduce unnecessary execution, but they also introduced a maintenance burden.
As the repository evolves, so must the rules. Moving files, introducing new components, or changing dependencies can require updating the path matchers and scripts that determine which workflows run.
We could continue adding rules and make them increasingly sophisticated, but that would also make them harder to understand and maintain.
Consider a pull request where the first commit changes the Kubernetes charm and its integration tests pass. The developer then pushes another commit that only changes the README.
If we evaluate the entire pull request, the Kubernetes charm is still among the changed files, so the integration tests run again.
If we only evaluate the latest commit, we can skip those tests. But what if the previous tests failed, were cancelled, or never ran?
We now need to consider not only which files changed, but also the previous CI results and whether they are still applicable.
All of this can be implemented with scripts and rules. However, the more cases we try to cover, the more complex those scripts become.
This made us consider a different approach.
From matching rules to making decisions
Large language models seemed like a natural solution. After all, the problem is about understanding a change, considering the available information, and deciding which workflows are relevant.
We could describe what each workflow does, provide the changes introduced by a pull request, and ask a model which workflows should run.
The problem is that using a general-purpose LLM to replace relatively simple Bash scripts would be difficult to justify, especially considering the cost and overhead of generating responses for every pull request.
This became more interesting with the emergence of models designed specifically for classification and decision-making.
Jev 1.14 is one such model. Instead of generating free-form text, it accepts structured information and questions with predefined choices. It returns a decision, probabilities for the available choices, and a confidence score.
This allows us to turn our workflow selection problem into a set of classification questions.
For each workflow, we describe its purpose, dependencies, and the conditions under which it should run. We then provide the pull request information and ask Jev whether that workflow needs to execute.
The important difference is that the decision criteria are expressed in natural language, making them easier to read and adjust without maintaining an increasingly complex set of path patterns.
Another interesting aspect is the cost. At the time of the experiment, OpenRouter listed Jev at $0.042 per million input tokens, with no output-token charge. We set a $1 spending limit to see how far that would take us.
We decided to experiment with two classifiers: one to select which CI workflows to run and another to decide whether failed tests should be retried.
1. Selecting which CI workflows to run
The first classifier runs when a pull request is opened or updated.
We collect the files changed across the pull request, the changes introduced by the latest commit, and the results of relevant CI jobs on the direct parent commit.
We also calculate whether previous validation can be reused. For example, if Kubernetes integration tests passed and the latest commit only modifies documentation, the previous results may still be applicable. If those tests failed or were cancelled, however, we cannot consider the change validated.
Jev receives this information along with a description of each workflow and its dependencies.
A simplified version of the request looks like this:
{
"model": "typesafe/jev-1.14",
"state": {
"pr_changed_files": [
"k8s/src/charm.py",
"README.md"
],
"commit_changed_files": ["README.md"],
"candidates": [{
"workflow": "k8s-integration-test",
"prior_validation_reusable": true
}]
},
"questions": {
"candidate_3": {
"type": "choice",
"instructions": "Should k8s-integration-test run?",
"criteria": {
"run": "The workflow needs new validation.",
"skip": "The workflow is irrelevant or prior validation still applies.",
"unknown": "There is insufficient evidence."
}
}
}
}
This example only includes a subset of the information we provide to Jev. The actual request contains more detailed workflow descriptions, dependencies, and decision criteria. You can find the complete implementation in Vault PR #1063.
The model evaluates the information and returns its decision, together with the probability and confidence.
Keeping the decisions under control
We don’t want an uncertain classification to cause important tests to be skipped.
For standard AMD64 workflows, running the tests remains the default. We only skip a workflow when Jev chooses skip with at least 80% probability and 80% confidence.
If the model is uncertain, the request fails, or the response is invalid, the workflow runs.
For additional ARM64 and s390x testing, we use the opposite default. These tests run when Jev confidently determines they are necessary or when a maintainer explicitly requests them.
Maintainers can also force individual workflows or the entire CI pipeline through labels.
These thresholds are configuration choices, not a guarantee of the model’s accuracy. Evaluating whether the decisions are correct is an important part of the experiment.
We also preserve full test coverage on the main and release branches, including additional architectures, before publishing artifacts.
The model is responsible for deciding which validation is relevant, while our workflow code remains responsible for collecting reliable information, enforcing dependencies, and applying the decisions.
2. Deciding when to retry failed tests
Selecting which tests to run is only one part of the problem. Another source of wasted time comes from intermittent integration test failures.
Our integration tests depend on environments that can occasionally fail for reasons unrelated to the changes being tested. For example, a network request might time out, Juju might take longer than expected to initialize, or Vault might not be ready when a test expects it.
These failures often require a developer to investigate the logs and manually rerun a job, sometimes hours after the original failure.
We wanted to automate this decision without blindly retrying every failure.
We introduced another Jev classifier with three possible answers: retry, do_not_retry, and unknown.
Our initial approach was to wait until a workflow completed, inspect the failed jobs, and use GitHub’s API to rerun them.
However, this required waiting for the entire workflow to finish and provisioning a new runner and environment for the retry.
We eventually moved the classifier inside the job itself.
The process is now straightforward:
Run the step → if it fails, ask Jev → retry the step once if the decision is confident.
When a step fails, a reusable GitHub Action sends the relevant failure logs and context to Jev.
If the model chooses retry with at least 80% probability and 80% confidence, we repeat the step once. Otherwise, the job fails normally.
There are no repeated retry loops, and successful steps do not require any model calls.
Lessons learned
Building these classifiers also required us to account for several details beyond the model itself.
Workflow dependencies matter. Integration tests depend on artifacts produced by build jobs. If a workflow is enabled, its prerequisites must also be available. The classifier’s decisions cannot be applied independently without considering these dependencies.
Previous CI results must be reliable. A cancelled test is not a passing test, and an incomplete commit comparison cannot prove that earlier validation still applies. We handle these checks in code before passing the information to Jev.
Not everything needs to be decided by AI. We initially used path filters to preselect candidate workflows before asking Jev. We eventually removed those duplicated patterns and let the model evaluate all five workflow groups using their descriptions and the available context.
However, deterministic checks remain responsible for gathering information and enforcing the rules around how decisions are applied. We also treat commit information and logs as untrusted data, rather than instructions for the model.
The goal is not to replace all our CI logic with AI, but to replace the parts where maintaining increasingly sophisticated decision rules becomes difficult.
What’s next? Measuring the results
This is still an experiment. We’ve focused on implementing the classifiers and testing their behavior, and our next step is to deploy them in Vault and evaluate whether they provide meaningful improvements.
We want to answer several questions.
Are we actually saving computing resources? We’ll compare the total GitHub Actions runner time before and after introducing the classifiers, taking into account the number of CI runs and the additional cost of model calls.
Are we missing important validation? Skipping tests is only useful if it doesn’t introduce regressions. One metric we’ll track is the number of follow-up fixes on the main branch caused by issues that should have been caught before merging. We’ll use this alongside the classifier’s decisions to investigate whether any skipped validation contributed to those issues.
Are pull requests getting merged faster? Intermittent CI failures can delay a pull request until a developer returns to investigate and rerun the failed tests. Automatically retrying likely transient failures might reduce these delays. Measuring this accurately is more challenging because merge times depend on many factors, but it is something we’d like to explore.
How much does the model actually cost? We also want to share how many classification requests were made, how much we spent, and how long our initial $1 budget lasted.
We’ll use these results to evaluate both the effectiveness and reliability of the approach, and share our findings in a follow-up article.
Conclusion
In this experiment, we’re exploring whether a small decision model can replace increasingly complex rule-based logic in our CI workflows.
The main advantage we’re looking for is not just reducing runner time, but also reducing the effort required to maintain the rules that determine which tests run and which failures deserve another attempt.
Whether this approach delivers meaningful improvements remains to be seen.