Your tests are grading the copy

Remember the telephone game from grade school? One kid whispers "the purple elephant is eating soup," and by the end of the line it has turned into "the principal is selling his boots."

Nobody lied. Every kid passed along exactly what they heard, and something still dropped out at every handoff. Nobody in the middle could tell, because each of them only ever compared against the kid right before them.

Most projects run the same game with more expensive kids, and the version I see most often has a row of green checkmarks at the end of the line.

The chain nobody draws

A website migration is the cleanest example I know, and if you have a website you will eventually do one.

The real thing is the live site: every URL it serves and every link on every page. Almost nobody works from the real thing directly. Someone runs a crawl and exports it to a spreadsheet, the spreadsheet becomes a redirect list, the redirect list becomes code, and then someone, increasingly an AI, writes tests to confirm the code covers the list.

That's four translations away from the source, and the tests check the last translation against the one before it. When they pass, they're telling you something true and nearly useless, which is that the copy agrees with itself.

Passing tests only prove the copy agrees with itself.

What fell out

I ran into this on a site rebuild this month, and I'd built half the chain myself, so I can't even blame a vendor. The redirect list came from a crawl export, every check was green, and the launch was close enough that I was mostly thinking about DNS.

Before cutover I ran the comparison against the actual live site instead, meaning its published sitemap plus the links rendered on every page. Three URLs fell out that the export never had. Two were menu folders, the kind of nav item a site builder creates that quietly redirects to its first child page. Folders like that never appear in a sitemap or a crawl export, and these two were linked from every page on the site, 84 times each. The third was a duplicate page someone had published after the crawl ran.

It was the second time that same export had missed live URLs. The first time, two blog posts that were ranking on Google got shorter slugs during the rebuild and the redirect list never heard about it. The fix was identical both times, which is how I knew the URLs were a symptom and the reference was the actual problem.

Why AI makes it faster and quieter

An AI will build from whatever you hand it, and it's very good at it. Hand it the spreadsheet and you get clean redirect code and thorough tests, every one of them pointed at the spreadsheet. It has no way to know the spreadsheet is a lossy copy unless you tell it what the original is.

So the telephone game speeds up. You can go from source to shipped in an afternoon with green checkmarks the whole way, losing fidelity at every step without a single red flag. The danger is confident, well-tested work against the wrong reference, and speed only makes it arrive sooner.

The boring fix that compounds

1. 1

   Name the canonical source, out loud.

   For a migration it's the live site, and your export of it is a copy. For a product it might be the contract, the customer's actual workflow, or the system you're replacing. Pick one, and write down that it's the one, so the next person (or the next model) doesn't quietly promote a copy to the original.

2. 2

   Write it down as a schema.

   This is the skill I'd tell anyone building with AI to get good at. A paragraph that says "make sure all the old pages redirect" gives a model nothing to check. A file where every URL has a path, a status, whether it's indexed, where it goes now, and why, gives a human and a model something to verify line by line. A good schema forces a decision for every row, and an empty field becomes a question you can see.

3. 3

   Keep comparing against the source every time you ask "are we done?"

   Copies drift, and the source is the only thing that tells you whether you're complete or just consistent. On that rebuild the URL list now lives in a file the test suite reads, so the next time the live site and the list disagree, the build fails instead of the launch.

If you have a website, the check takes about an hour. Pull the live sitemap, crawl the links on the rendered pages (menu folders and duplicate pages hide there), diff both against your redirect plan, and commit the result as data your tests read. From then on, that file is the source.

Skipping it costs you the URLs Google already trusts, discovered one 404 at a time by your customers.

The purple elephant is still eating soup. Go check.