Too Good to Be True? The Precognition Paradox
On this page
Are psychology studies reporting suspiciously perfect results?
Imagine you're reading a scientific paper claiming extraordinary results - maybe evidence for telepathy or a revolutionary new treatment. The findings look impressive: study after study showing significant effects. But here's the twist: a researcher named Ulrich Schimmack discovered that when papers report too many 'successful' experiments, they might actually be less trustworthy, not more. His analysis revealed a statistical paradox that challenges how we evaluate scientific evidence.
Research articles may be reporting too many positive results to be statistically believable.
In 2012, psychology was facing a credibility crisis. High-profile studies were failing to replicate, and researchers were questioning whether published findings could be trusted. Statistician Ulrich Schimmack decided to investigate whether there was mathematical evidence of problems in how psychology studies were being conducted and reported.
When scientific papers report too many significant results across multiple studies, it may actually signal questionable research practices rather than strong evidence.
Key Findings
- Many psychology articles reported far more statistically significant results than would be mathematically expected given their study designs.
- This pattern suggested that researchers might be using questionable practices to artificially inflate their success rates, such as selectively reporting only positive findings or manipulating analyses until they found significant results.
What Is This About?
Schimmack analyzed published psychology papers that contained multiple experiments, focusing on how many statistically significant results they reported. He calculated what percentage of positive results each study should have found based on their statistical power (their ability to detect real effects). He then compared this to how many positive results were actually reported, using controversial studies like Bem's ESP research and Gailliot's glucose studies as examples.
Statistical analysis of published research articles to examine whether they report more significant results than would be expected given their statistical power.
Found that multiple-study articles often report suspiciously high numbers of significant results, suggesting potential questionable research practices.
How Good Is the Evidence?
This paper has been cited 470 times, reflecting its significant impact on psychology's methodological reforms. The statistical techniques Schimmack developed are now widely used to detect potential research misconduct.
Methodological reformers argue this analysis reveals widespread problems in psychology research and validates concerns about publication bias and questionable research practices. Traditional researchers counter that the statistical assumptions may be flawed and that legitimate research strategies could explain some apparent anomalies. Both sides generally agree that increased transparency and pre-registration would benefit the field.
Mainstream: This analysis demonstrates serious methodological problems that require systematic reforms in research practices. Moderate: The statistical patterns are concerning but may reflect a combination of publication bias and legitimate research decisions rather than misconduct. Frontier: These techniques could be overly rigid and might discourage exploratory research that leads to genuine discoveries.
Misconception: This study proves that ESP research is fake. Reality: This is a methodological analysis that uses ESP studies as one example of suspicious statistical patterns - the same problems were found across many areas of psychology, not just parapsychology.
To validate these concerns, we'd need large-scale replication studies of suspicious findings, analysis of researcher practices through surveys and interviews, and demonstration that pre-registered studies show different patterns than non-pre-registered ones. This study provides the statistical foundation for such investigations and has indeed prompted many of these follow-up studies, contributing to psychology's ongoing methodological reforms.
The discrepancy between the expected number of significant results and the actual number of significant results in multiple-study articles undermines the credibility of the reported results, and it is likely that questionable research practices have contributed to the reporting of too many significant results.
Stance: Mixed
What Does It Mean?
The most fascinating aspect is that Schimmack turned statistics into a detective tool - using math to uncover when research results are suspiciously perfect. It's like having a lie detector for scientific papers.
It's like a basketball player claiming to make 90% of their free throws, but when you calculate based on their skill level, they should only make 60%. The gap suggests something fishy might be happening - maybe they're only reporting their best games.
If Schimmack's analysis is robust, it suggests that some highly cited research - including studies on controversial topics like ESP - may be built on shakier foundations than previously thought. This could reshape how we evaluate evidence in psychology and other fields, potentially leading to more rigorous standards for publication and replication. The implications extend beyond individual studies to questions about the reliability of entire research literatures.
When reading research, look for whether authors explain how they determined their sample size beforehand - studies that seem to have 'just enough' participants to reach significance might be suspicious.
Understanding Terms
What This Study Claims
Methodology
Multiple studies with modest statistical power have a high probability of producing nonsignificant results because power decreases as the number of tests increases
strongInterpretations
The discrepancy between expected and actual significant results in multiple-study articles undermines credibility of reported results
moderateQuestionable research practices have likely contributed to reporting too many significant results in psychology journals
moderateImplications
Authors should justify sample sizes with a priori predictions of effect sizes to increase credibility
moderateReplication studies with nonsignificant results should be published if they have high power to replicate a published finding
moderateThis summary is for general information about current research. It does not constitute medical advice. The scientific interpretation of these results is debated among researchers. If personally affected, please consult qualified professionals.