The invisible half

Every experiment rests on a question you can't answer: what would have happened if we hadn't done the thing?

This hypothetical – the outcome in the world without intervention – is called the counterfactual. It's the invisible half of every experiment. You can never observe it, only construct it. And how you construct it determines whether your conclusions are sound or fiction.

Getting it wrong is easy. A recurring part of my work is finding where someone built the wrong counterfactual and showing why their conclusions collapse.

The same error runs outside data too, into decisions where no one audits the comparison.

A confident wrong answer

A company runs an ad campaign. Budget constraints mean only 40% of the eligible audience actually sees it. The other 60% are in the experiment pool but never get the ad.

When measuring whether it worked, the tempting shortcut: compare the exposed group against all of control. Treatment on the left, control on the right. The exposed group bought more. Campaign worked.

Except the exposed 40% didn't get there randomly. They're systematically different from the unreached majority: more active, more likely to be in-market, more likely to convert regardless. Comparing them against an unfiltered control doesn't measure the campaign's effect. It measures self-selection wearing the campaign's name.

The fix is precise: if it's not like for like, it's not a valid comparison. Everything else is just a mirror angled to confirm what you already believed.

A bad counterfactual gives you a confident wrong answer. That's worse than no answer at all.

The Silicon Valley trap

The tech mythology runs like this: the dropout who built a billion-dollar company at 19, the engineer who turned down Google at 22 and made something from scratch. These stories are real. They're also a broken counterfactual for anyone who uses them as a benchmark.

The reason isn't talent – it might be equal, might not be, and you can't measure it cleanly enough to know. What you can measure is starting position. And starting position is what you inherited before the gun went off, not when you entered the race.

There's a tailwind that accumulates invisibly when you grow up inside an ecosystem: the university network that quietly routes opportunities your way, the childhood friend who joined a fund, the professor who makes an introduction, the cultural literacy that means you know how rooms work before you ever enter one. Nobody earned these things. They're interest that compounds on being born in the right place. The safety net that makes failure survivable is capital, not courage.

When you arrive mid-life from somewhere else, you're not starting at zero. You're starting negative, working to close a gap that others never had to think about. The skills travel. The tailwind doesn't come with them.

Which means the comparison group you've been measuring against was never valid.

A sample size of one

Counterfactual reasoning works when you can find a valid comparison group: random assignment, similar distributions, comparable starting conditions. The more your conditions diverge, the less any external benchmark tells you.

Stack enough divergent variables and the pool of valid comparisons shrinks.

At some point it reaches one.

When n=1, there is no control group. There never was. Every comparison you've been running – every time you measured yourself against the mythology – was already invalid. Not directionally off, but structurally broken. The number was never measuring what you thought it was.

The method that holds is pre-post: measuring against your own prior state. Not am I where they are? but am I further than I was? It strips away the noise of incomparable starting conditions and measures the only quantity you have a clean read on: your own trajectory.

From where you actually started, are you gaining?