The AI Models Smart Enough to Know They're Cheating — Beth Barnes & David Rein [METR]

107 snips

May 4, 2026

Guest

David Rein

Guest

Beth Barnes

David Rein, researcher behind GPQA and METR time-horizons work, and Beth Barnes, METR founder and former OpenAI alignment researcher, discuss measurement of AI capabilities. They unpack the METR time-horizon graph, benchmark pathologies, reward hacking and agent harnesses. They debate scaffolding, verifiability, labor-market effects, and how to interpret timelines without overclaiming.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

Avoid Adversarial Benchmarks To See Stable Trends

Adversarially selected benchmarks (like ARC) tend to regress then rapidly be saturated by labs, causing volatile performance spikes.
METR deliberately avoids adversarial selection to reduce regression-to-mean and get steadier trends.

ANECDOTE

First Time An Agent Recognized Its Own Process

Early agents would kill their own process or misidentify running processes, but newer models began recognizing "that one's me" in process lists.
Beth Barnes recounted the first time a model inspected running processes and correctly identified its own process.

ADVICE

Tell Agents Their Token And Time Budget

Tell agents their token/time budget explicitly during runs to improve calibration and prevent premature or excessive work.
Beth Barnes found models misallocate effort without token-usage signals, so add runtime budget cues to scaffolds.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Beth Barnes and David Rein on the one graph that ate the AI timelines discourse, and why the two people who built it are the most careful about how you read it.**SPONSOR**Prolific - Quality data. From real people. For faster breakthroughs.https://www.prolific.com/?utm_source=mlstInterview: https://youtu.be/cnxZZTl1tkk---Beth Barnes and David Rein from METR on the one graph that ate the AI timelines discourse, and why the people who built it are the most careful about how it gets read.Beth founded METR after leaving OpenAI alignment. David is first author on GPQA and co-author on HCAST and the METR Time Horizons paper. Together they built the measurement Daniel Kokotajlo called the single most important piece of evidence on AI timelines: the log-linear line of "how long a task a frontier model can complete at 50% reliability" vs release date.The conversation opens on reward hacking. Current models can articulate in chat why a behaviour is undesired and then execute it anyway as agents. From there: construct validity, Melanie Mitchell's four-problem taxonomy, and the ARC-AGI 1-to-2 collapse as a worked example of adversarially-selected benchmarks regressing once labs target them. Beth's counter: METR deliberately does not adversarially select. David's: models do not have to do the right thing for the right reasons.Methodology, then specification — David's compiler analogy, Beth on four-month tasks as expensive to evaluate rather than unspecifiable. Then the SWE-bench reality check, the METR finding that half of passing PRs would not be merged, and Beth's horses-versus-bank-tellers analogy for the labour market.The close: monitorability, the coin-spinning boat, two-year recursive self-improvement, and Beth's line that "overhyped now" and "big deal later" are not correlated claims.---TIMESTAMPS:00:00:00 Intro00:02:06 Sponsor break: Prolific human-feedback infrastructure00:02:33 Welcome and the scalable oversight motivation00:06:02 Construct validity, benchmark pathologies and the Chollet worry00:15:45 Time Horizons: human time, HCAST tasks and the 50% logistic00:24:50 Is human difficulty really one variable?00:33:05 Agent harness evolution and the inference-compute dividend00:40:00 Scaffolding bells, token budgets and the credit-assignment problem00:44:15 Look at the damn graph: regularisation bug and reliability nuance00:50:00 Why 50%? Reliability, reward hacking and pizza-party transcripts00:55:20 Extrapolation risk and straight lines on graphs00:59:25 Software engineering as a specification acquisition problem01:07:40 Compilers also made ugly code: vibe-coding quality and Claude on METR Slack01:15:15 Strongest defensible claim, Carlini's compiler swarm and AI 202701:23:45 SWE-bench merge rates, the bank-teller analogy and horses01:31:45 Scheming, alignment faking and the mentalistic vocabulary problem01:40:45 Reward hacking, monitorability and chain-of-thought faithfulness01:45:25 Recursive self-improvement, knowledge vs intelligence and closing

ReScript: https://app.rescript.info/public/share/de3bb40cc02ee39fdf36e2c60366eb4d

(PDF, refs, transcript etc)