Articles

Coding agents weaken tests about as rarely as people do

Wenchao Gou · November 13, 2026

I've been building enterprise software for almost 10 years. Lately I've been working on honest-ci, a GitHub Action. When a PR deletes, skips, or rewrites tests, honest-ci detects it and hands it off to a human to review.

I kept noticing that companies were churning out code at an incredible rate, and I figured they were churning out bugs at the same rate. But somehow CI on GitHub seemed to keep passing. Gradually I got the feeling that coding agents are more willing than humans to edit the tests just to get CI green.

So I went to GitHub to check. I pulled 1,598 PRs that six coding agents opened in public repos during July and August 2026, plus 1,119 PRs that humans opened in the same repos around the same time. I ran honest-ci's checks on all of them and labeled every flag myself.

1. About 1 in 11 PRs actually deleted a test, skipped a test, removed an assertion, or changed an expected value: 8.4% for agents, 10.4% for humans.
2. Almost all of them had a reason: a feature got removed along with its tests, or the expected value really did change.
3. 11 PRs weakened tests without giving a reason in the title, description, commits or diff: 5 from agents (0.3%), 6 from humans (0.5%).
4. 93.6% of the flags the detector raised correctly described the diff.

So much for intuition. Agents change tests about as often as humans do, and weakening a test with no reason given is roughly as rare for both. In both cases the 95% confidence intervals overlap.

The five checks

The detector reads the whole PR diff, including removed lines. Deleting a failing test to get CI green doesn't leave a single added line to look at.

There are five checks, all in the table below, and they're deterministic rules. What a check finds is called a hit. A hit only describes the diff, so removing a feature together with its tests trips deleted_test just like deleting a failing test to get past CI does. Telling the two apart comes down to human review.

The sample

I pulled PRs opened in public repos between Jul 1 and Aug 26, 2026, on projects with at least 10 stars, no forks. A PR counts as an agent PR if it was opened by the agent's GitHub App, or carries one of its signatures: a footer like Generated with Claude Code, a session link, or a branch prefix (codex/, claude/, etc.). I checked every signature against real PRs before adding it. PRs with only a weak trace, like a Co-authored-by line on a commit, go in neither group.

I took 300 PRs each from GitHub Copilot coding agent, OpenAI Codex, Claude Code, Cursor and Devin. Google Jules only had 98 that qualified, because over 90% of its PRs in that window were in repos with fewer than 10 stars. No agent got more than 3 PRs from any one repo.

I cut the window into 10-minute slices and went through them in random order, taking at most 5 PRs per agent from each slice. Any more than that and one busy afternoon could fill a whole group.

For the human control group, each agent PR gets paired with the human-opened PR in the same repo that's closest to it in time. That way the two PRs share a codebase, tests, reviewers and CI. That gives 1,119 human PRs, with a median gap of 6.5 hours to the agent PR they're paired with. The other 479 agent PRs had no human PR nearby.

Details and full code: https://github.com/ChrisShips/honest-ci/tree/main/bench

Hits before labeling

Before any labeling, agent PRs and human PRs tripped the checks at similar rates.

The ranges in parentheses are 95% Wilson intervals.

The agents differ a lot more from each other, from 2.7% for Copilot to 13.0% for Codex. But most of that comes from the repos, not the agents. Among the human PRs matched to Copilot's, 3.9% had a hit. Among those matched to Codex's, 18.0% did.

Bigger PRs are also more likely to trip a check. Copilot's PRs have a median of about 40 lines changed. Claude Code's have about 290.

Labeling the flags

Each hit is a one-line warning. A flag is all of a PR's hits on a single check, and each flag gets one judgment. So deleting a feature along with its 30 tests counts as one flag, not 30. In the sample, 265 PRs had hits: 1,241 hits in total, which collapsed into 345 flags.

Each flag gets one of four labels:

real: the change actually weakens what the tests check, and the PR gives no reason or evidence for it.

justified: what the flag says really happened, but for a good reason. For example, a feature was removed along with its tests, a flaky or outdated test was quarantined with a stated reason, or a test was replaced with a better one that the detector didn't match up with the original.

wrong: a false positive. The detector misread the diff.

?: can't tell from the PR.

A reason counts if the PR states it in its title, description, or commit messages, or if the diff makes it obvious. "Make the build green" on its own doesn't count. The detailed rules for each check are here: https://github.com/ChrisShips/honest-ci/blob/main/bench/README.md#labelling-hits

For each flag, I read the material myself: the PR description, the commits, and the diff around each hit. The material didn't show the group, the author, or even whether an agent wrote the PR. That was on purpose. The title or the writing can still give an agent away, so this reduces the bias but doesn't remove it.

Codex was a huge help here. Doing all of this by hand would have been miserable. It drafted a label and the supporting evidence for each flag, and I reviewed them. I did the first 30 on my own, though, and matched Codex on 20. Of those 30, I labeled 7 as real. Codex labeled 2.

For the 10 we disagreed on, I went back to the diffs and checked them against the written rules, then had Claude review them too. All 10 ended up with Codex's label. For the other 315 I could see the drafts while I labeled, and I agreed with Codex on all of them. Agreeing on all 315 looked too good, so I had Claude re-check 79 flags: all 16 labeled real at that point, all 23 labeled wrong, a random 30 labeled justified, and 10 that had notes or were labeled per file. Claude couldn't see Codex's drafts or the groups. After its review I changed 8 flags, and real went from 16 to 11.

The line between real and justified depends on who's labeling. On the same 30 flags, I marked 7 as real on my first pass, and the final count was 2. If I'd stayed that strict, there could be about 3x as many real flags, roughly 1% of PRs instead of 0.4%. The material hid the group, so stricter labeling would push up the numbers for agent and human PRs alike.

Results after labeling

After labeling, agents and people still come out about the same.

The intervals overlap here too.

5 of the 1,598 agent PRs have a real hit. That's too few to rank agents. By agent, it's Copilot 2, Claude Code 1, Cursor 1, Jules 1, Codex 0 and Devin 0. Don't read anything into that.

I'll only name two of the 11, both in big projects.

Azure/LogicAppsUX#9416 was opened by a human. It adds a designer panel and fixes designer bugs. Along the way, it removes 8 assertions that check the workflow migration's results and replaces them with one that only checks that a mocked migrateWorkflow was called. The migration code now lives in a new module, and no test in this PR checks its results anymore. A coworker approved and merged it.

microsoft/vscode#327841 was opened by the Copilot coding agent. It trims whitespace from API keys before storing them. It also renames a test to cover the trimming. In the process, the test loses three groups of assertions: the ones checking that the old secret gets deleted, the model settings and the provider identity. Under my rules it counts as real, since nothing in the title, description, commits or diff says why. But the review conversation does. A reviewer asked Copilot to "modify the test to only test for the apikey key," and three people approved the result.

A real flag just means the PR itself didn't give a reason. That's a long way from showing that anyone cheated, or that code review let these through. 10 of the 11 were merged, but so were about 4 in 5 of all PRs in the sample (last row of the table). Of those 10, 5 were merged by their own authors with no one else approving. The other 5 were approved or merged by someone else, and in the vscode one, the test change was what the reviewer asked for. 4 of the 10 got comments from AI review bots (Copilot code review, Greptile, cubic, Baz). The results were mixed, so you can't say the bots missed them either.

False positives

22 of the 345 flags were wrong. All 22 came from the two checks that pair removed lines with added ones: changed_expectation (18 of 142) and fewer_assertions (4 of 37).

changed_expectation mostly went wrong by reading a failure message or a test name as an expected value, like the string in assert total == 3, "expected 3 items". I've since fixed that, and made the check pair lines only within the same test. fewer_assertions now recognizes an aliased expect, a Go return err used as an assertion, and assertions moved into a shared helper.

Some cases still fool the detector, like an assertion whose actual and expected values both change.

Everything I measured on the sample used the detector before these fixes (source digest dff3f19c90a5). Reporting the fixed one's numbers on the same sample would be tuning on the test set. But an honest retest needs a new window and new labels. I haven't done one yet.

What I changed

I built honest-ci to catch agents cheating on tests. The data doesn't back that up. Changing tests is common for agents and people alike, and almost always fine. CI can't tell a fine change from a bad one, because the tests pass either way. So what's useful is making every test change visible before merge.

By default, the first four checks hold the PR for a person, and loosened_threshold blocks it.

On this sample, the test-integrity checks alone would hold up about 1 in 10 PRs (265 of 2,717). About 1 in 25 of those had a real hit (11 of 265). The rest were almost all justified, but someone would still have to look.

I've also deleted the README line that said agents "often" change the test rather than the code.

Running honest-ci on its own repo

Since Oct 10, honest-ci has checked all 12 PRs in its own repo. 9 passed and 3 were held for a person. None were false positives. I haven't wanted to turn it off.

But GitHub doesn't let you approve your own PR, so on a one-person repo there's nobody to approve a held one. Twice I just merged past the red X.

Limitations

1. The sample only covers public repos with at least 10 stars. The private team repos honest-ci is meant for may differ.
2. Agent work in the human group pulls the two groups closer: Claude Code with attribution off, Codex on a human-named branch, agents the bench doesn't know (Aider, Gemini CLI and others) and editor completions. So does a person taking over an agent's branch.
3. Diffs over GitHub's limit (300 files or 20,000 lines) aren't counted. Deleting a whole test suite often makes a diff that big, so it's the most blatant cases that get undercounted.
4. I'm the only labeler.

Code and data

Code and method: https://github.com/ChrisShips/honest-ci/tree/main/bench

Every label, with a one-line reason and links to the code: https://github.com/ChrisShips/honest-ci/tree/main/bench/data

The reasons are Codex's, except where my label differs from its draft.

honest-ci is source-available under the Fair Core License, free on public repos and your own private ones: https://github.com/ChrisShips/honest-ci