Replication crisis and Kahneman's studies

What happens when you discover that a book that fundamentally changed how you think is built on a shaky foundation? In today's episode, I share my own struggle with the replication crisis surrounding Daniel Kahneman's *Thinking Fast and Slow*, and I use it as a springboard to talk about a much bigger skill: knowing how to update your beliefs when reality shifts underneath you. This isn't about throwing out science or losing trust in your heroes. It's about developing the muscle to replace old explanations with better ones — a skill that has never been more important for software engineers.

• The Replication Crisis, Briefly Explained: Understand the difference between reproducing a study (re-running the analysis on the original data) and replicating one (recreating the study from the ground up), and why a surprisingly large portion of well-respected psychology research, including studies cited in Thinking Fast and Slow, doesn't hold up under scrutiny.
• Base Rates Matter: Kahneman didn't pick uniquely bad studies. If you randomly sampled from the broader academic literature, you'd hit the same failure rate. The lesson isn't about one author — it's about how we evaluate any body of knowledge.
• The Beginning of Infinity Framework: Drawing from David Deutsch's book, explore the idea that all progress is rooted in the assumption that we are fundamentally incorrect, and that improvement comes from continually building better explanations on top of incomplete ones.
• Beliefs as Calibration, Not Truth: Your beliefs about what makes a good engineer, what makes good code, or what makes a good career move are not eternal truths. They are calibrations to your current reality, and that reality is changing fast.
• The Ego Trap of Old Beliefs: Notice the very human, very subtle pull to defend things you previously argued for — not because they're still right, but because admitting otherwise creates a discontinuity with your former self. This is one of the biggest blockers to learning.
• Two Competing Explanations of AI Adoption: Walk through a worked example of holding two predictions about AI in tension and asking honestly which one better explains the reality you're seeing — at both a macro industry level and the micro level of debugging a system.
• Moving Goalposts Aren't a Conspiracy: A lot of what feels like shifting goalposts in our industry is just goalposts moving on their own. A big part of our job as engineers is figuring out where they are now and predicting where they're heading next.
• Episode Homework: Pick one belief you hold strongly about your work — about what makes a good engineer, about a tool, about a process. Try to deconstruct it into its parts and ask whether a better explanation exists for what you're actually seeing.

🙏 Today's Episode is Brought To you by: SerpApi

No matter what you're building, SerpApi is the web search API for your needs. If you're building an application that needs real-time search data—whether that's an AI agent, an SEO tool, or a price tracker—SerpApi handles it for you. ● Make an API call and get back clean JSON. ● They handle the proxies, CAPTCHAs, parsing, and all the scraping so you don't have to. ● They support dozens of search engines and platforms, and are trusted by companies like NVIDIA, Adobe, and Shopify. ● If you're building with AI, they even have an official MCP to make getting up and running a simple task. Get started with a free tier to build and test your application before you commit. Go to serpapi.com.

📮 Ask a Question

If you enjoyed this episode and would like me to discuss a question that you have on the show, drop it over at: developertea.com.

📮 Join the Discord

If you want to be a part of a supportive community of engineers (non-engineers welcome!) working to improve their lives and careers, join us on the Developer Tea Discord community today!

🗞️ Subscribe to The Tea Break

We are developing a brand new newsletter called The Tea Break! You can be the first in line to receive it by entering your email directly over at developertea.com.

🧡 Leave a Review

If you're enjoying the show and want to support the content head over to iTunes and leave a review!