Red-green TDD with AI agents: the test must fail first

AI agents write tests that pass. That is not the same as tests that prove anything, and a senior who has to tell the two apart in review loses the time AI was supposed to save. So our gate runs every new test against the old code first. If it passes there too, the PR is red.

This is red-green-refactor, enforced by a machine instead of discipline. AI writes all of our code today, and a senior is accountable for the result. Red-proof is how that senior can trust a green check.

Why are AI-written tests a risk?

Ask an agent to "add tests" and you get tests that go green. Left alone, it tends to write them after the code, against the code, so they confirm what the code already does. Some do not even call the code. We have seen tests that read a source file as text and looked for a function name. They pass on any version that contains that word.

Coverage goes up. Proof stays where it was. The reviewer becomes the last line, and reading every test closely is exactly the work you hoped to stop doing.

Where does your team stand?

What is red-proof?

The red step of red-green-refactor, checked by a script:

1. Red. The new test must fail on the base branch, the code before the change.
2. Green. The same test must pass on the branch.
3. No test, no change. Source changed without a test changed is a failure.

The brief our agent gets for code tasks says the same in words. Write one test for the riskiest criterion and commit it alone. It must fail on the unmodified code. Then implement. Do not edit the test after its commit.

Words are a request. The gate is the lock.

How does the gate check it?

It is plain bash, gate.sh in our agent harness. The core, shortened:

# red-proof: the branch's new/changed tests must FAIL on the base code, or they prove nothing
changed=$(git diff --name-only "$base"...HEAD; git ls-files --others --exclude-standard)
tests=$(printf '%s\n' "$changed" | grep -E '(^|/)(tests?|__tests__|spec)/|\.(test|spec)\.[a-z]+$')
src=$(printf '%s\n' "$changed" | grep -E '\.(ts|tsx|js|jsx|mjs|py|go|rs|php|rb)$' \
      | grep -vE '(^|/)(tests?|__tests__|spec)/|\.(test|spec)\.')
if [ -z "$tests" ]; then
    [ -n "$src" ] && echo "FAIL red-proof: source changed but no test was added or changed"
else
    cmd=$(jq -r .scripts.test package.json)
    git worktree add -q --detach "$W/base" "$base"      # the old code, next to the branch
    for t in $tests; do cp "$t" "$W/base/$t"; done      # the new tests, onto the old code
    if (cd "$W/base" && sh -c "$cmd") > "$W/red-test.log" 2>&1; then
        echo "FAIL red-proof: the new tests also pass on $base, so they do not prove the change"
    else
        echo "PASS red-proof: new tests fail on $base and pass on the branch"
    fi
fi

Three rules sit around it:

• Tests without a test script fail. Nothing would run them.
• The reviewer reads why the tests failed on base. The gate prints the first assertions from the red run. A failure from a missing file or import is not a red proof, and the reviewer says so.
• Tests that read source as text go back. The reviewer's instructions name them: a test must call the function, render the component or request the route.

The gate does not mean more tests are better. We delete pointless tests. Sometimes the fix is to simplify, not to add.

What broke: a false pass on a red main

The first version had a hole. When the base branch already failed its own test suite, every new test looked red there. The gate reported PASS for tests that proved nothing.

The fix: if the base suite already fails, run only the new test files on their own. On base they must fail, on the branch they must pass. If they cannot run on their own, the result is inconclusive, and inconclusive is a failure.

scoped=""
(cd "$W/base" && sh -c "$cmd") > "$W/base-test.log" 2>&1 || scoped=" $(echo $tests)"

The second hole was the environment. On one of our projects, red-proof ran the tests without the right setup: a tool was missing (turbo: command not found) and a different Node version ran. The tests on base failed before they even started. The gate took that as red proof, so every red-proof there was a false pass.

The same fix tightened screenshots. A visible UI change used to pass with any PNG attached. Now it needs a before and an after.

Does every change need a red test?

No. A pure refactor or a config change has nothing new to prove. Everything else goes through the gate:

The opposite holds for critical software that calculates prices or decides about people. There we never switch the gate off.

How do you add red-proof to your CI?

The same gate is in our public Claude Code production starter, MIT licensed. Run it locally before you say done:

bash scripts/gate.sh all origin/main

The order we recommend:

The starter carries the simpler version without the scoped rerun, so keep your main branch green. That is good advice anyway.

Rules like "a test must fail first" belong in the repository, not in a senior's head. More in Rules for AI code in a team. Why the checks around the model matter more than the model is in Harness engineering in practice: from issue to PR.

Want tests from AI that prove something?

We set up gates like this right in a team's repository as part of AI-native SDLC for development teams. Is your team short of hands rather than rules? See a senior developer for your team. Since 2023 we have delivered 15+ production projects. A senior plugs into your repository and pull requests on day one.

We get back to you within two business days, and together we pick what to fix first.

If you'd rather talk first, book a 15-minute meeting with Filip.

Which of your tests would still pass if you deleted the feature?

Frequently asked questions

What is red green refactor?

A test-driven development cycle: write a test that fails (red), write the code that makes it pass (green), then clean up (refactor). The red step proves that the test checks the new behaviour.

How do you do test-driven development with AI agents?

Have the agent write one test first and commit it alone, then implement. Enforce it in CI: the new test must fail on the base branch and pass on the branch. Otherwise the PR is red.

Why must a new test fail on the base branch?

A test that passes before and after the change does not test the change. It may test nothing at all, or only behaviour that already existed.

What if the main branch already has failing tests?

Then the whole suite fails on the base branch and every new test looks red. Run only the new test files on their own. If they cannot run alone, treat the result as a failure, not a pass.

Can an agent skip red-proof?

Not on its own. A pure refactor or a config change can be exempt, but a person decides that, with a label on the pull request.