Models Change Every Week, But Nobody's Reviewing: Why Verifying AI Code Beats Picking the Right Model by 10x

TL;DR

Over the past two years I’ve used nearly every Coding Agent out there: Claude Code, Codex, Cursor, Gemini CLI, OpenClaw… the models rotate and the leaderboards refresh daily. But I’ve grown convinced of one thing: what determines whether you succeed with AI isn’t which model you pick — it’s whether you have a real skill for verifying the code AI writes. The model generates the code; verification decides whether it ships. The former is now fast and cheap; the latter has quietly become the tightest bottleneck in the whole pipeline. This post covers how that bottleneck formed, the 6 most common traps in AI-generated code, and the four-layer acceptance system plus a desk checklist I actually use.

Figure 1: Hero image — AI churns out PRs on a conveyor belt 24/7 while a human inspects them one by one with a magnifier. Self-made.

1. Background: We All Caught “Model Selection Anxiety”

Confession: I used to be a hardcore “model person.”

Every new model release, I was first in line — checking benchmarks, scrolling review threads on X, then throwing the same task at three or four agents to compare whose code was more elegant, whose reasoning felt more “like me.” I even maintained a “model-to-task mapping table”: use A for refactors, B for tests, C for debugging, D for copy edits.

Sounds professional, right? Until one day I was doing a post-mortem on a production incident and noticed something embarrassing: the code that broke was generated by what was then the top-benchmarked model. And the reason it broke wasn’t that the model wasn’t smart enough — it was that I saw it “looked plausible,” skimmed it, and merged.

That moment it clicked: I had been optimizing the production side nonstop, while investing zero in the acceptance side. It’s like a restaurant owner who keeps upgrading to a better robot chef but never checks whether anyone tastes the food before it goes out. No matter how fast the kitchen cooks, if nobody tastes it, sooner or later something goes wrong.

2. Symptoms: Acceptance Is the New Traffic Jam

Once you start using a Coding Agent, a subtle role change happens: you go from “the person who writes code” to “the person who reviews code.”

You used to write 200 lines a day; now the agent spits out 2000 lines in an hour. The generation step got compressed to minutes, but the step of “understanding what those 2000 lines actually do and whether I dare merge them” didn’t compress at all — it got stretched by sheer volume. The pipeline’s bottleneck quietly migrated from production to acceptance.

Figure 2: The bottleneck shift — AI compresses generation to minutes while unreviewed code piles up. Self-made diagram.

This isn’t just my feeling. In June 2026, The New Stack published a piece with a telling title: How long before we stop reading the code? Its core argument: AI is “breaking” traditional code review, and the human checkpoint must move upstream — from reading code to reviewing intent, specs, and acceptance criteria.

Figure 3: Real screenshot of the New Stack piece. Note the subtitle — “Stop reading code and move the human checkpoint upstream to review intent, specs, and acceptance criteria.” Even industry media is talking about this bottleneck.

In plain terms: there’s too much code and not enough eyeballs. You can no longer read every line the AI writes — you have to verify differently. The problem is, most people’s verification method is still stuck at the primitive stage of “glance at it, seems fine, merge.”

3. Analysis: You Feel Faster, But You’re Actually Slower

You might push back: “I clearly feel faster with AI — how is acceptance the bottleneck?”

Fair question. And it’s exactly the most counterintuitive, most alarming part of the whole thing. Let’s look at some hard data.

In July 2025, the research org METR ran a rare randomized controlled trial (RCT — the same gold-standard method medicine uses to test drugs). They recruited 16 experienced open-source developers and had them complete 246 real tasks on mature projects they knew extremely well (averaging 5+ years of maintenance experience, over a million lines of code), randomly split into “AI allowed” and “AI not allowed.”

Figure 4: Real screenshot of the METR paper on arXiv. A serious 51-page study with 8 tables and 22 figures — not some hot-take blog post.

The results shocked everyone:

• Before the experiment, developers predicted AI would make them 24% faster;
• After the experiment, they still felt they had been 20% faster;
• But the measured result was that they were actually 19% slower.

Figure 5: Feeling vs. measurement — a full 39-percentage-point gap. Self-made chart, data from the METR paper.

Perception and measurement diverged by 39 points, in opposite directions. Why? Digging in, METR’s researchers found that when developers used AI, huge amounts of time were eaten by three things: iterating on prompts, reviewing AI-generated code, and fixing the subtle bugs AI introduced. And developers only accepted 39% of AI suggestions — meaning more than 60% of AI output was discarded after a human looked at it.

See it? The time saved didn’t vanish — it just moved from “writing” to “verifying.” And because “verifying” is so fragmented and tedious, your brain never registers how long it takes, producing the illusion that “I got faster.”

Here’s an analogy a kid could follow: it’s like ordering takeout. Delivery takes 5 minutes (AI generation), but once it arrives you spend 40 minutes checking whether they sent the wrong dish, forgot the chopsticks, or spilled the soup (human verification) — then you still have to reheat it and add salt before eating (fixing bugs). You saved “cooking” time, but “inspecting the delivery” ate all of it back, plus interest. You thought you ordered a convenient meal; you actually worked a free 40-minute shift as a quality inspector.

4. Root Cause: Code Is Undergoing “Bad Money Drives Out Good”

If it were only “no time saved,” that would be tolerable. Worse: AI is quietly changing the quality structure of entire codebases.

GitClear is a company that analyzes code quality in git repos. They analyzed 211 million changed lines from 2020 to 2024 (across repos from Google, Microsoft, Meta, and more), and in their 2025 code quality report found some sobering trends:

Figure 6: Moved/refactored code fell from 25% to under 10%, while copy-pasted code overtook it. Self-made chart, data from GitClear.

• The share of “moved/refactored” code dropped from ~25% in 2021 to under 10% in 2024. Moved code is a classic signal of refactoring — extracting duplicated logic for reuse. Its collapse means people are increasingly not “tidying the house.”
• 2024 became the first year in history where “copy-pasted code” exceeded “moved code.” The number of duplicated code blocks grew 8x in 2024.

Translation: AI is great at “writing something new,” but terrible at “making existing code better.” It would rather copy-paste a fifth near-identical copy of a logic block than refactor the first four. Because “copying” costs it nothing, while “refactoring” requires understanding the whole project’s context — which is precisely its weakest spot.

This is why “acceptance” matters more than “which model.” A renovation analogy: AI is a construction crew with superhuman strength and speed, but zero understanding of your floor plan. Ask it to “add an outlet,” and it’ll tear down and rebuild the whole wall in three minutes — and throw in three extra walls while it’s at it (copy-paste). The walls go up fast and straight, but your home gets more chaotic and less livable. At that point, what determines whether your home stays comfortable isn’t how strong the crew is (how good the model is) — it’s whether you, the supervisor, can yell “stop” when it starts building that third unnecessary wall.

Even Simon Willison (co-creator of Django and one of the most influential independent voices on AI-assisted coding) coined a distinction: he calls the “don’t read the code, if it runs it ships” style vibe coding, and the “professional stays deeply accountable for AI output” style vibe engineering (the industry now often says “agentic engineering”).

Figure 7: Real screenshot of Simon Willison’s famous “Vibe engineering” post. He explicitly says: using agents to produce “production-quality code I’m confident I can maintain” is an entirely different way of working.

The line between these two isn’t how expensive your model is — it’s whether you take responsibility for, and verify, the output.

5. The Fix: My Four-Layer Acceptance System

Enough problem-talk — here’s the solution. This is a four-layer acceptance system I settled on after stepping on countless rakes. The core idea borrows from airport security — layered checkpoints, each catching a different class of problem, stricter the deeper you go.

Figure 8: The four-layer system — like airport security, machines check what machines can, human eyes are reserved for what matters most. Self-made.

Layer 1: Intent Acceptance — Did it deliver the dish I ordered?

When you get the AI’s output, the first thing isn’t reading code — it’s reading “what it thinks it did.” Look at its summary, the commit message, the list of files in the diff, and ask one question: is this change the same thing I asked for?

AI’s most common vice is “over-delivering” — you ask it to fix pagination, and it rewrites your whole list component plus throws in three “optimizations” you never wanted. Intent acceptance catches these “smuggled extras” at the door. It’s like checking the tracking number before opening a package — if the number’s wrong, it doesn’t matter how nicely it’s wrapped.

Layer 2: Automated Acceptance — Never hand-check what a machine can check

Tests, lint, type checks, security scans, builds — all automated, all hard gates for merging. The principle here: whatever a machine can judge should never consume precious human attention.

But there’s a huge trap, which I’ll come back to: AI-written tests can be fake. So this layer needs one extra step beyond “run the tests”: “verify the tests.”

Layer 3: Manual Acceptance on the Critical Path — Spend good steel on the blade’s edge

Human attention is the most expensive resource — spend it only where failure is unaffordable. What’s unaffordable? Code touching money, data, permissions, security. That code gets read line by line by hand, no matter how pretty the AI made it look.

This is like airport security: every bag goes through the X-ray (automation), but valuables and suspicious packages get opened and hand-inspected (manual). Hand-checking everything will exhaust you; hand-checking nothing will get you hurt.

Layer 4: Rollback Acceptance — Leave Yourself an Exit

The last line of defense, and the most overlooked. No matter how carefully you verified, assume you missed something. So keep every merge small, independent, and cleanly revertable. That way, if a bug does slip through to production, you can roll it back in 5 minutes and keep the damage tiny.

The ultimate goal of acceptance isn’t “never make mistakes” — it’s “even when a mistake happens, find it fast, revert it fast, and walk away unharmed.”

6. The 6 Traps in AI Code: A Bug-Hunting Checklist

A system isn’t enough — you also need to know what the “bugs” look like. I’ve distilled the most common problems in AI-generated code into 6 traps. Run down this list every time you verify.

Figure 9: The 6 common traps in AI code, with a review mantra. Self-made.

1. Looks-runnable hidden pit: passes locally, but shatters on empty arrays, concurrency, timezones, integer overflow. Like an umbrella that works in sunshine but leaks in a downpour.
2. Off-target scope creep: you asked for A, it did A plus rewrote B, C, and D — the change far exceeds the ask. You ordered noodles; the chef renovated the kitchen.
3. Security backdoor left wide open: hardcoded keys, auth “temporarily commented out for debugging,” validation switched off. The contractor didn’t bother installing a door lock.
4. Copy-paste duplication: the same logic cloned 5 times; fix one, miss four. That’s exactly where GitClear’s 8x came from.
5. Style drift: naming, layering, and error handling are a completely different dialect from the rest of the project — foreign code that doesn’t belong.
6. Fake tests going through the motions: the sneakiest one — assertions that always pass (assertTrue(true)), failing cases commented out, “green” for green’s sake. The thermometer can’t read a fever, so it smashes the thermometer.

7. A Checklist to Tape to Your Desk

Finally, the whole system compressed into one checklist. Before merging any AI-written code, tick down the list — all ✅ before you merge.

Figure 10: The desk acceptance checklist — screenshot and save it. Self-made.

Two easily-missed items worth highlighting:

How do you verify “the tests are real”? There’s a brutal trick: deliberately break the code under test and see whether the tests go red. If the tests stay green after you break the code, they’re decorative — fake. It’s a poor-man’s version of “mutation testing,” and it’s devastatingly effective against AI’s fake tests.

Why is “I can explain it” the bottom line? Because the person signing the merge is you, not the AI. When production breaks, your boss comes to you, not the model. If you can’t explain why the code is written this way, then you weren’t really “verifying” — you were “drawing lots.” If you can’t explain it, don’t merge it — your signature, your responsibility.

8. Q&A: A Few Frequently Asked Questions

Q1: So is AI coding useless?

Quite the opposite. AI is enormously useful — provided “fast production, strict acceptance.” The METR study focused on senior developers doing fine-grained maintenance on hyper-familiar mature projects — precisely the scenario where AI gains the least and verification costs the most. In looser-acceptance scenarios — prototypes, throwaway scripts, exploring new domains — AI’s speedup is very real. The key is matching the scenario: don’t confuse “can generate quickly” with “safe to merge.”

Q2: Should I stop chasing new models?

Not stop — down-weight. The difference between model generations is far smaller than the difference between “has an acceptance system” and “doesn’t.” A mid-tier model + strict acceptance beats a top-tier model + blind merging. A model going from 80 to 90 points might give you a sub-10% felt improvement; but going from “glance and merge” to a “four-layer system” can cut your incident rate by an order of magnitude. The ROI isn’t even in the same league.

Q3: What if my small team has no headcount for four layers?

Four layers doesn’t mean four people. One person can do it: intent acceptance = “read the AI’s summary before opening the diff”; automated acceptance = “wire tests and lint into CI as required checks”; manual acceptance on the critical path = “force yourself to read anything touching money or permissions line by line”; rollback acceptance = “don’t yolo, split into small commits.” Low cost, big payoff.

Q4: Will models eventually be so good that acceptance is unnecessary?

Maybe someday. But not today, and not within the next few visible years. The stronger the model, the more important the tasks we dare hand it — and the bigger the verification stakes. It’s like seatbelts: the better the car and the faster it goes, the less you can skip the belt. Acceptance skill isn’t a transitional skill for the AI era — it’s a core skill of it.

Closing

Back to the title’s question: why does “how to verify AI-written code” matter more than “which model to pick”?

Because the model is a variable; acceptance is a constant. Models rotate generation by generation — today’s #1 is tomorrow’s has-been. But “do you stand behind every line you merge” never goes out of date.

AI turned “writing code” into a cheap commodity — and in doing so, pushed the price of judgment through the roof. When everyone can generate a thousand lines in three minutes, what truly separates people is the one who knows which three of those thousand lines can kill you.

So stop staring at the leaderboard. Go train your acceptance skill — that’s the moat in the AI coding era that no one can take from you.