2024-03-22: AI Test Automation
how quickly is this going to happen
🔷 Subscribe to get breakdowns of the most important developments in AI in your inbox every morning.
Here’s today at a glance:
🦾 Automate It All
Meta put into production an AI code generation tool for test code, and this paper describes what they did.
The system is an LLM-driven unit test generator. Once cases are generated, they are:
confirmed to be buildable; avoiding instances of LLM hallucination errors
confirmed they pass; failing tests may be failing because of a mistake in the test, or because of a bug in the code. Since they are targeting automatic code generation, they do not bother to find out which, simply discarding failing tests
confirmed they pass at least 5 times; to avoid “flaky“ tests that only sometimes fail
confirmed the addition of the test increased test coverage; they need a quality metric to judge the test worthy of addition to the test suite, and this is it.
A test that passes all the filters is a useful o…
