Reward hacking as the core failure mode

Geoffrey argues diverse bad behaviors are manifestations of long-standing reward-hacking under optimization pressure.

Play episode from 01:21:32

chevron_right

Transcript

chevron_right

Transcript

Episode notes

Geoffrey Irving, Chief Scientist at the UK AI Security Institute, explains why our theoretical understanding of machine learning remains fragile even as models surpass experts on critical security tasks. He details AISI’s work on frontier model evaluations, red teaming, and threat modeling across biosecurity, cybersecurity, and loss-of-control risks. The conversation explores reward hacking, eval awareness, and why current safety techniques may struggle to deliver high reliability. Listeners will also hear how AISI is funding foundational research to build stronger guarantees for AI safety.

Nathan uses Granola to uncover blind spots in conversations and AI research. Try it at ⁠granola.ai/tcr⁠ with code TCR — and if you’re already using it, test his blind spot recipe here: ⁠https://bit.ly/granolablindspot⁠

Sponsors:

Serval:

Serval uses AI-powered automations to cut IT help desk tickets by more than 50%, freeing your team from repetitive tasks like password resets and onboarding. Book your free pilot and guarantee 50% help desk automation by week 4 at https://serval.com/cognitive

Claude:

Claude is the AI collaborator that understands your entire workflow, from drafting and research to coding and complex problem-solving. Start tackling bigger problems with Claude and unlock Claude Pro’s full capabilities at https://claude.ai/tcr

Tasklet:

Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai

CHAPTERS:

(00:00) About the Episode

(04:09) From physics to ML

(08:52) AGI uncertainty and threats (Part 1)

(18:08) Sponsors: Serval | Claude

(21:29) AGI uncertainty and threats (Part 2)

(27:35) Control, autonomy, alignment (Part 1)

(34:02) Sponsor: Tasklet

(35:14) Control, autonomy, alignment (Part 2)

(38:44) Inside the UK AC

(51:02) Evaluations and jailbreaking

(01:01:17) Emerging capabilities and misuse