What are Just-in-Time Tests (JiTTests)?
Just-in-Time Tests are automatically generated test cases created on the fly for a specific code change. Unlike traditional static test suites, JiTTests are produced by large language models (LLMs) at the moment a pull request is opened, aiming to catch regressions before the code reaches production.
How JiTTests Work
The JiTTest workflow can be broken into distinct steps:
- New code change is detected (e.g., a pull request).
- The system infers the developer's intent behind the change.
- Mutants—versions of the code with injected faults—are generated to model possible failures.
- LLMs synthesize test cases that target the inferred intent and the created mutants.
- Generated tests are executed against the mutants.
- Rule‑based and LLM‑based assessors filter results, highlighting true‑positive failures.
- Engineers receive a concise report of unexpected behavior, without needing to write or review test code.
Why JiTTests Are Transforming Testing
Agentic development accelerates code turnover, making traditional testing a bottleneck. JiTTests address three core challenges:
- Maintenance Overhead: Tests are not stored in the repository, eliminating ongoing updates.
- False Positives: By tailoring tests to the specific change and using intent inference, JiTTests reduce noise that distracts engineers.
- Speed: Automated generation and execution happen instantly, keeping pace with rapid development cycles.
Comparison: Traditional Testing vs. JiTTests
Traditional testing relies on manually authored, static test suites that must be maintained as the code evolves. JiTTests differ in several key ways:
- Static vs. Dynamic: Traditional tests are static; JiTTests are generated per change.
- Human effort vs. Machine effort: Traditional tests require continuous human authoring; JiTTests shift effort to LLMs.
- Broad coverage vs. Targeted coverage: Traditional suites aim for general coverage; JiTTests focus on the specific intent and potential regressions of a change.
Benefits of Deploying JiTTests
Organizations that adopt JiTTests can expect:
- Reduced test maintenance costs.
- Higher signal‑to‑noise ratio in test results.
- Faster feedback loops for developers.
- Improved ability to keep testing in step with AI‑driven, agentic development practices.
Considerations and Best Practices
While JiTTests offer many advantages, teams should be mindful of:
- LLM reliability: Validate generated tests periodically to ensure quality.
- Security and privacy: Ensure code snippets sent to LLM services do not expose sensitive information.
- Integration: Embed JiTTest generation into existing CI/CD pipelines for seamless operation.
- Human oversight: Reserve manual review for cases where a test flags a potential bug.