Episode 66: Superforecasting

whatshot 28 snips

Mar 4, 2025

Explore the intriguing world of superforecasting, where accurate predictions come from understanding probabilistic language. Dive into Cold War tensions that shaped critical decision-making moments and learn why teamwork may not always enhance forecasting accuracy. Discover the nuanced skills that distinguish exceptional forecasters, while reflecting on the dangers of flawed research in Alzheimer's studies. Embrace open-mindedness and the power of Bayesian reasoning to refine beliefs and improve predictive capabilities.

01:15:53

forum

Ask episode

web_stories

AI Snips

view_agenda

Chapters

menu_book

Books

auto_awesome

Transcript

info_circle

Episode notes

question_answer

ANECDOTE

Experts' Predictions about Gorbachev

The US National Research Council formed a panel to discuss preventing nuclear war.
Philip Tetlock observed how experts' biases shaped their interpretations of Gorbachev's appointment.

insights

INSIGHT

Expert Predictions vs. Chance

Tetlock's Expert Political Judgment study revealed that experts' predictions are often no better than chance.
He emphasized falsifiable, time-limited predictions to avoid vague verbiage.

insights

INSIGHT

Brier Score and Calibration

The Brier score assesses prediction accuracy by considering the squared error of forecasts.
It heavily penalizes highly confident, incorrect predictions, encouraging careful calibration.

Get the Snipd Podcast app to discover more snips from this episode

Cold War Tensions and Predictions

01:44 • 11min

chevron_right

Understanding Probabilistic Language and Forecasting

12:42 • 10min

chevron_right

The Art of Superforecasting

23:06 • 13min

chevron_right

Unmasking Flawed Research

36:22 • 4min

chevron_right

Exploring the Art of Forecasting and Improvement

39:58 • 2min

chevron_right

The Psychology of Forecasting

41:31 • 14min

chevron_right

Embracing Open-Mindedness and the Outside View

55:38 • 2min

chevron_right

Forecasting Success: Marriages and Divorce Rates

57:14 • 4min

chevron_right

Bayesian Reasoning in Superforecasting

01:00:48 • 15min

chevron_right

#1770

• Mentioned in 24 episodes

The Scout Mindset

Julia Galef

Dans "The Scout Mindset", Julia Galef explore la différence entre deux approches mentales : le "Mindset du Soldat", axé sur la défense de ses croyances, et le "Mindset du Scout", qui privilégie la recherche de la vérité, même si cela remet en question ses propres convictions. L'auteure argumente que le Mindset du Scout est essentiel pour une prise de décision rationnelle et efficace. Le livre propose des outils et des techniques pour cultiver ce Mindset, en encourageant l'ouverture d'esprit, la remise en question et l'acceptation de l'incertitude. Galef utilise des exemples concrets et des anecdotes pour illustrer ses points, rendant le livre accessible à un large public. L'ouvrage est une invitation à adopter une approche plus objective et plus rigoureuse dans la formation de ses opinions et dans la prise de décisions.

#4571

• Mentioned in 11 episodes

Doctored

Charles Piller

#4585

• Mentioned in 11 episodes

Radical Uncertainty

Decision-Making Beyond the Numbers

John Kay

Mervyn King

In 'Radical Uncertainty: Decision-Making Beyond the Numbers,' Mervyn King and John Kay discuss the concept of radical uncertainty, which involves events whose determinants are insufficiently understood for probabilities to be known or forecasting to be possible. The authors argue that many critical decisions cannot rely on forecasts or probability distributions and instead advocate for strategies that are robust to alternative futures and resilient to unpredictable events. They critique the overreliance on statistical models and rational choice theories, emphasizing the importance of narratives, abductive reasoning, and understanding the context of decision-making. The book covers various fields including economics, finance, policy studies, and more, using real-life examples to illustrate the folly of predictive models and the need for a more nuanced approach to uncertainty.

#499

• Mentioned in 61 episodes

Superforecasting

The Art and Science of Prediction

Dan Gardner

Philip E. Tetlock

Santiago Foz (argentino)

In this book, Tetlock and Gardner present the results of the Good Judgment Project, a massive forecasting tournament that identified a small group of 'superforecasters' who are exceptionally good at predicting future events. The authors explain that good forecasting does not require powerful computers or arcane methods but involves gathering evidence from various sources, thinking probabilistically, working in teams, keeping score, and being willing to admit error and change course. The book uses stories of forecasting successes and failures, as well as interviews with high-level decision makers, to illustrate these principles and demonstrate how anyone can improve their forecasting abilities.

#852

• Mentioned in 42 episodes

On Bullshit

Harry Frankfurt

In 'On Bullshit', Harry G. Frankfurt argues that bullshitters misrepresent themselves to their audience not by deliberately making false claims, but by conveying a certain impression without concern for whether anything is true. He distinguishes bullshitting from lying, noting that liars at least acknowledge the importance of truth, whereas bullshitters disregard it entirely. Frankfurt contends that excessive indulgence in bullshit can undermine the capacity to tell the truth in a way that lying does not, making bullshit a greater enemy of truth than lies. The book explores the prevalence of bullshit in modern society and its implications for communication and truth.

#12947

• Mentioned in 4 episodes

Everything Is Predictable

How Bayes’ Remarkable Theorem Explains the World

George Orwell

#3179

• Mentioned in 15 episodes

The Logic of Scientific Discovery

Karl Popper

Karl Popper's "The Logic of Scientific Discovery" is a highly influential work in the philosophy of science. It critiques the traditional view of scientific method as inductive, arguing instead that scientific knowledge is developed through a process of conjecture and refutation. Popper introduces the concept of falsifiability, suggesting that a scientific theory must be testable and potentially falsifiable to be considered scientific. The book has had a profound impact on the philosophy of science, influencing the way scientists approach the development and testing of theories. It remains a cornerstone of modern epistemology.

Whether it’s the 1903 New York Times article that claimed a flying machine was ten million years away, or the record executive who (allegedly) told the Beatles in the early 1960s that guitar bands were on the way out, predictions are hard.

In this episode of The Studies Show, Tom and Stuart discuss the psychologist Philip Tetlock’s research on superforecasters, the people who make the most accurate predictions of all. Even if you can’t become a superforecaster yourself, it turns out there’s a lot we can learn from them about how to form beliefs—and how to be right more often.

The Studies Show is brought to you by Works in Progress magazine, where this week Tom has written a review of the new book, Doctored, about fraud in Alzheimer’s research. Read that and many other short pieces on the Works in Progress Substack at worksinprogress.news.

Show notes

* A book chapter on the “Expert Political Judgement” study from Philip Tetlock

* Research on how people interpret terms like “a serious possibility” and “likely”

* Research that argues against the idea that teaming up makes superforecasters better

* Study on the correlates of being a good superforecaster (i.e. having a low Brier score)

* A paper on “small steps to accuracy”: how people who update their beliefs more often are better forecasters

* Philip Tetlock and Dan Gardner’s book Superforecasting

* Julia Galef’s book The Scout Mindset

* Tom’s book, Everything is Predictable

* Tom’s review of Mervyn King’s book, Radical Uncertainty

Credits

The Studies Show is produced by Julian Mayers at Yada Yada Productions.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit sciencefictionspod.substack.com/subscribe

Home Top podcasts Popular guests Top books