correlation isn’t causation even when you worked hard and made it good
Most everyone who has taken stats or studied policy or listened at some point to a TED talk has run into the truism “correlation is not causation.” It has become so commonplace that I usually find myself annoyed when I hear someone say it. Yeah, no duh, get outta here with your annoying cliches.
But it can be really easy to forget this when your team has a) worked really hard on something and b) made that thing as good or better than most things in its reference class. When conditions a) and b) are met, it’s really super tempting to say, believe, and bet that the best-in-class thing your team built is the cause of the results you’re seeing.
It’s even more tempting when you’ve got squishy, shaky, or thin data. The things you know for sure (we busted tail on this piece; this piece is now good) can overwhelm your analysis of what this piece is actually doing for you.
A quick list of things that might make you assume causation when you don’t have evidence beyond correlation:
An elaborate hiring process (we have great people! It must be because of this hiring process)
A complicated feature of your app or product (we have so many new users using our thing so much more! It must be because of this feature)
A costly security protocol (we have so few incidents! It must be because of this onerous protocol)
A mystique and bureaucracy can build up around the best-in-class thing your team built. As you grow and go, it’s worth vetting any expensive process or feature you’ve built in the past. Is this thing really driving your results? Or has it become an opera house - beautiful, costly to maintain, a reminder of what we like about ourselves, but sort of inviolate and untouchable and possibly not the reason people visit our city?
-eric
