Science Fictions Episode 93: Many analysts
19 snips
Jan 13, 2026 Discover the intriguing Many-Analysts Problem, where identical datasets yield wildly different conclusions. Delve into examples from sports data highlighting bias, and explore how researchers' choices shape findings in studies. The discussion spans fMRI results, ideological influences, and the impact of data cleaning on outcomes. With critiques of traditional analysis and suggestions for improvement, this episode invites listeners to rethink the objectivity of science. Join the conversation on transparency, collaboration, and what it means for research credibility!
AI Snips
Chapters
Books
Transcript
Episode notes
Wide Outcomes, Little Explained Variation
- A 2022 PNAS many-analysts study found huge variation: teams supported, rejected, or deemed the hypothesis untestable.
- The authors said methodological choices and beliefs explained little of that variance.
Borjas Reanalysis Finds Ideology Signals
- Economist George Borjas reanalyzed the 2022 PNAS study and found ideological effects.
- He argued prior beliefs shaped methodological choices and thus outcomes.
Effect Sizes Often Tell A Calmer Story
- Focusing on statistical significance exaggerates perceived disagreement.
- In the immigration study most effect estimates clustered near zero despite differing p-values.
