During World War II, Allied bomber losses were high, so the powers that be demanded a fix. The engineers set out to eyeball every bomber they could, gathering great statistics for each bullet hole. After a long study they decided to add more armor plating to the areas that had the highest concentrations of holes. A bit after these improved planes were deployed, they received some startling news: more planes were going down than before. At this point I thought, “Did they make them too heavy?”
Then the light bulb went on for someone: they had measured every bullet hole in every plane at their disposal, but they’d failed to realize it was the ones that they did not have access to that mattered. It was the ones that did not return that needed to be scrutinized. They needed to improve the armor in the places that the returning planes had no bullet holes.
Sounds like it might generally relate to product marketing, and user studies: go investigate the customers that don’t return for seconds, i.e. upgrade. After all, if I don’t return to buy another product from you or upgrade say my first generation product to your newer product, you gotta find out why. Let’s see how many organizations actually to do this!
