The Invisible Half

Every experiment rests on a question that sounds deceptively simple until you try to answer it: what would have happened if we hadn’t done the thing?

This hypothetical, the outcome in the world without intervention, is called the counterfactual. It’s the invisible half of every experiment. You can never observe it, only construct it. And how you construct it determines whether your conclusions are true or fiction.

Getting it wrong is easy and more common than you’d think. An important part of my work is finding where someone built the wrong counterfactual and explaining why the conclusions don’t hold.

The same error shows up outside data, in places where no one runs the audit.

A Confident Wrong Answer

A company runs an ad campaign. Budget constraints mean only 40% of the eligible audience actually sees it. The other 60% are in the experiment pool but never get the ad.

When measuring whether it worked, the tempting shortcut: compare the exposed group against all of control. Treatment on the left, control on the right. The exposed group bought more. Campaign worked.

Except the exposed 40% didn’t get there randomly. They’re systematically different from the unreached majority: more active, more likely to be in-market, more likely to convert regardless. Comparing them against an unfiltered control doesn’t measure the campaign’s effect. It measures the effect of self-selection, not the campaign.

The fix is precise: if it’s not like for like, it’s not a valid comparison. Everything else is just a mirror angled to confirm what you already believed.

A bad counterfactual doesn’t just give you the wrong answer. It gives you a confident wrong answer. Which is worse.

The Silicon Valley Counterfactual Trap

The tech mythology runs like this: the dropout who built a billion-dollar company at 19, the engineer who turned down Google at 22 and made something from scratch. These stories are real. They’re also a broken counterfactual for most people, and the numbers bear that out.

The reason isn’t talent: it might be equal, might not be, and you can’t measure it cleanly enough to know. What you can measure is starting position. And starting position isn’t just when you entered the race. It’s what you inherited before the gun went off.

There’s a tailwind that accumulates invisibly when you grow up inside an ecosystem: the university network that quietly routes opportunities your way, the childhood friend who joined a fund, the professor who makes an introduction, the cultural literacy that means you know how rooms work before you ever enter one. Nobody earned these things. They’re not merit. They’re just the interest that compounds on being born in the right place. The safety net that makes failure survivable isn’t courage. It’s capital.

When you arrive mid-life from somewhere else, you’re not starting at zero. You’re starting negative, working to close a gap that others never had to think about. The skills travel. The tailwind doesn’t come with them.

Which means the comparison group you’ve been using was never valid to begin with.

A Sample Size of One

Counterfactual reasoning works when you can find a valid comparison group: random assignment, similar distributions, comparable starting conditions. The more your conditions diverge, the less any external benchmark tells you.

Stack enough divergent variables and the pool of valid comparisons shrinks.

At some point it reaches one.

When n=1, there is no control group. There never was. Every comparison you’ve been running, every time you measured yourself against the mythology, was already invalid. Not directionally off. Structurally broken. The number you were getting was never measuring what you thought it was.

The method that holds is pre-post: measuring against your own prior state. Not am I where they are? but am I further than I was? It removes the noise of incomparable starting conditions entirely and measures the only quantity you have a clean read on: your own trajectory.

From where you actually started, are you gaining?

That’s the only valid question.