Into AI Safety Sobering Up on AI Progress w/ Dr. Sean McGregor
8 snips
Dec 29, 2025 
Dr. Sean McGregor, a machine learning safety researcher and founder of several initiatives like the AI Incident Database, delves into the complexities of AI evaluation. He critiques the flaws in current benchmarking practices, emphasizing their vulnerability to training-data leakage and real-world misalignment. Sean introduces BenchRisk, a new framework aimed at improving benchmark trustworthiness. He also discusses the founding of AVERI, a nonprofit focused on frontier model auditing to ensure responsible AI deployment and navigate the tension between market and regulatory safety.
AI Snips
Chapters
Transcript
Episode notes
Built An Incident Database From Scratch
- Sean founded the AI Incident Database to catalog real-world AI harms and prevent repeats.
- He modeled it on aviation and product-safety databases to enable collective learning.
Benchmark As A Risk Management Tool
- Treat benchmarks as risk-management artifacts and design for longevity and monitoring.
- Benchrisk recommends documenting scope, assumptions, and evergreen monitoring to avoid misleading users.
Claims Must Match Benchmark Design
- Benchmarks must align claimed use-cases, data, and metrics or they'll mislead deployment decisions.
- Sean emphasizes clear claims and matching data for domain-specific risks like arson prompts.