How is it made sure that something has been discovered, and not just noise? Is one discovery of something that is predicted considered to be enough (Higgs-particle)? What are the probabilities of a false positive in current experiments, like the dark matter experiments? How do they compare to the probability of a true positive?
Asked
Active
Viewed 519 times
3
-
See the variance and sigmas. Each result is announced with a variance, meaning its credibility , ie 6 sigmas for the Higgs – May 30 '15 at 22:27
-
2https://xkcd.com/882/ – Steeven May 30 '15 at 22:31
-
2That's a very broad question... you are already getting answers regarding statistics, but that's not even half the rent. More importantly in a high quality experiment we have to understand all predictable systematic error sources and they all have to be characterized by control experiments. Only if systematic AND statistical errors are fully understood AND we have at least a second independent experiment (of similar quality) that can also reproduce the effect can we be reasonable sure that we have found something new. – CuriousOne May 30 '15 at 22:33
-
1On a practical note, I have used the Median Filter on several experiments. It is an incredible "cheat", and remarkably simple. You take (say) 10 readings, put then in numerical order than throw away the outliers ie the minima and maxima. Never been quite sure how legitimate it is though. – May 31 '15 at 15:26
2 Answers
10
This is something that particle physicists are perfectly well aware of. For any given observed effect, there is always a nonzero probability that the observation will be a false positive that was caused by a random fluctuation. The name of the game is taking enough data that this probability is small enough. In general, the more data you take, the less likely it is that statistical fluctuations will persist.
If a group wants to claim the discovery of a new particle, the golden standard is five sigma: the size of the effect must be five times its standard deviation away from what you'd expect if the particle didn't exist. If an experiment satisfies this condition, then there is a chance of 1 in 35 million that if the particle didn't exist, the data would look like do by sheer statistical fluke. If this probability is any bigger, the particle physics community holds off on making the discovery official until the probability crosses this threshold.
Emilio Pisanty
- 132,859
- 33
- 351
- 666
-
2It might also be worth mentioning that in order to qualify as a discovery, the hypothesis that the particle does exist has to not be excluded at a statistically significant level. It's possible to have both models excluded at five sigma - though to be fair, that means you did discover something, just not what you were looking for. – David Z May 31 '15 at 09:55
-
@David that's some complicated mental gymnastics. But yeah, that's a good point. – Emilio Pisanty May 31 '15 at 12:05
-
Statistical significance (p-Values) and false positive probabilities (conditional probabilities) are not the same thing however. So are classical significance tests considered sufficient for declaring a discovery? – jjack May 31 '15 at 12:20
-
DavidZ will be in a much better position than me to answer that question, I'm afraid. – Emilio Pisanty May 31 '15 at 12:23
-
1@jjack you might actually need someone involved in detector data analysis to give a proper answer. I'm a "lowly" theorist, and I don't know the details of how the analyses are done. But in the presentations I've seen, everything is presented in terms of statistical significance. – David Z May 31 '15 at 15:34
-
1@DavidZ had a nice answer http://physics.stackexchange.com/questions/186274/how-do-tracking-detectors-in-particle-accelerators-create-the-pretty-pictures-we/186290#186290 about detectors - much of the conditional probability is in calibrating for the characteristics of these detectors, which must be taken into account during the reconstruction process. – paisanco May 31 '15 at 15:41
4
No, a single detection is not sufficient to conclude something new has been discovered. Enough events have to be observed to pass a statistical hypothesis test to reject the null hypothesis that what is being observed is indeed noise. As @EmilioPisanty stated in his answer, a "5 sigma" significance means that the observation is such that the chance of a false positive is equal to the area under the probability distribution at greater than 5 standard deviations from the mean, which is a very small probability.
Particle physicists have a very good idea of what the "noise" for a given experiment will look like in the detector, from experience with previous experiments. This "noise" consists, for instance, of signatures from other particle decay channels than those of interest that are caused in the course of the experiment, as well as various sources of electronic noise in the detector. A complex computer simulation of the detector and its response to noise and expected signal signatures, known as a Monte Carlo simulation, is a major part of the effort in any particle physics experiment.
The test of whether a discovery is made is not just a simple p-value test of the final mass or lifetime data of signal and background. Extracting a signal from the raw data is a multistage process, carried out with computer software- elementary particle physics is the original "Big Data" problem and had been since long before the term gained currency in business.
The conditional probabilities are taken into account during the reconstruction of the raw data from the detectors, incorporating both the characteristics of the detectors (probability of detection/false alarm, sources of systematic error) and the branching fractions of the many intermediate decay channels from the original particle to what is observed passing through the detectors. The original particle (Higgs etc.) decays into other particles, which in turn decay into other particles which decay into things like kaons, muons, protons which are long-lived enough to be seen passing through the detector.
paisanco
- 2,311