To understand how this fallacy applies to particle physics, it helps to consider the way the experiments work. Physicists don’t discover new particles the way scientists might find a new species of dinosaur. They’re too small to see and they last just a fraction of a second before disintegrating into a spray of other particles.
Those other particles leave tracks in a detector, but the tracks don’t come labelled. Deciding whether an unusual confluence of tracks came from a new particle is a little like determining whether leaks from a chemical plant might be increasing cancer risk among the local population. Some regions will have more cancer cases than others by chance. But the more cases that turn up in the region in question, the less likely an observed cluster is the result of chance alone.
It might prompt some worry if researchers calculated that there was only a 1-in-93 chance that, absent any carcinogenic chemical spill, a small town would see two dozen cases of some type of leukemia over a short time. But physicists aren’t dealing with sick people. For them, waiting is the prudent reaction to uncertainty.
They’ve been burned before. In 1984, the New York Times ran a story about a possible game-changing particle called the Zeta. Don’t worry if you haven’t heard of it. It never existed.
For doctors and lawyers, it’s useful to know the rarity of a disease or the fact that just one person in London committed a particular crime. Physicists have no equivalent information in estimating the odds that their new particle exists. Cousins said that all they can do is get a lot more data and wait until they see something that’s much more unusual – a pattern in the data with less than a one-in-a-million chance of appearing if there’s no new particle.
That level of caution is not so counterintuitive if you translate the problem to a poker game. Imagine you’re playing a seven-card game and one player says he’s been blessed by a fairy godmother who will help him win. Then he proceeds to get a full house. The odds, in a seven-card game, are about 37-to-1 without divine intervention. If he claims that his good hand proves there’s only a 1-in-37 chance his fairy godmother doesn’t exist, he’s fallen prey to the prosecutors’ fallacy.
Most sensible players wouldn’t buy the fairy godmother hypothesis even if this player beat 1-in-594 odds to get four of a kind. It’s easier to see the fallacy when a claim is outrageous.
Now, let’s say the player says he’s going to channel his fairy godmother to get a royal flush in a five-card hand. If he gets those cards, even savvy players start to suspect some kind of intervention, and not the divine sort. The odds of this happening by chance alone are not far from those it would take to convince the physicists of a new particle – about 650,000- to-1.
In a world ruled by uncertainty, how do scientists know when to declare a discovery? Cousins said that it’s a subjective decision, depending on how easy it is to collect more data, whether a scientist is in danger of being scooped and how much someone’s reputation will be hurt by being wrong. Certainty is always a gamble.