Module 12 Aggregate - Weigh and pull together the evidence

Send us a message with your thoughts or reactions

This episode was updated in 2025 to reflect changes to the online course relating to how best to aggregate the evidence (via parallel or serial approaches) and the importance of cross-validation with the parties contributing to the evidence.

This episode accompanies Module 12 of the course, which explores how to bring together the various sources of evidence gathered throughout the process. It’s the stage where the question becomes “What does this really tell us?” and where ideas about confidence, belief, and openness to new evidence come to the fore. The episode looks at why existing beliefs can be so sticky, how confidence shifts as new information arrives, and how a little Bayesian thinking can help keep our perspectives flexible.

There are also practical stories from the field, including how asking “How certain are you?” — or even framing a claim as a bet — can reveal far more than expected. The purpose of an evidence-based approach is to reduce uncertainty in decision making by examining likelihoods and probabilities, and this episode explores both how Bayes’ rule can support that and what to do when evidence appears to conflict. Contradictory evidence turns out to be far rarer than many students assume, and the discussion highlights how confidence levels can be surfaced and constructively challenged, and how cross-validation helps build shared understanding and ownership.

Aggregation is reframed not as a technical exercise but as a human one: a process of dialogue, reflection, and sense-making. The episode considers how to handle myths and “zombie ideas,” and how to craft an evidence story that is both accurate and memorable. Above all, the message is to slow down, check assumptions, and involve others — because good decisions depend on understanding the evidence together rather than rushing to action.

Further reading / sources mentioned during the episode:

• Nate Silver - "The Signal and the Noise"

Host: Karen Plum

 Guests:

• Eric Barends, Managing Director, Center for Evidence-Based Management 
• Denise Rousseau, H J Heinz University Professor, Carnegie Mellon University

Additional material with thanks to:

• Julia Galef - President and co-founder of the Center for Applied Rationality - YouTube videos 

Find out more about the course here:   https://cebma.org/resources-and-tools/course-modules/