Motivation for COT Interpretability

TYPE III AUDIO explains why reading chains of thought helps safety and why richer tools are needed beyond human reading.

Play episode from 00:32

chevron_right

Transcript

chevron_right

Transcript

Episode notes

Authors: Daria Ivanova, Riya Tyagi, Arthur Conmy, Neel Nanda

Daria and Riya are co-first authors. This work was done during Neel Nanda's MATS 9.0. Claude helped write code and suggest edits for this post.

TL;DR

One of our best safety techniques right now is “just read the chain of thought”.
But this isn’t always enough: can we learn more by going beyond just reading the reasoning?
Yet it's such an effective technique that it's hard to tell if we have made much progress on improving methods.
To help the community develop more powerful chain of thought analysis tools, we introduce and open source nine objective tasks, where a black box GPT 5.2 monitor falls short OOD.
We also baseline probes (linear, attention, SAE) and text frequency analysis (TF-IDF), and find they often do better than zero-shot and few-shot LLM monitors OOD. Methods that are useful in practice must be useful out of distribution, such that they’re not just learning spurious confounders in a poorly constructed distribution. As such, all tasks are evaluated on both an in-distribution and out-of-distribution test set.
When someone makes a better CoT interpretability method, we hope our testbed will help them prove that it works.

[...]

---

Outline:

(00:31) TL;DR

(02:09) Heres a list of our tasks:

(03:12) And the methods we test:

(04:46) Aggregate results

(05:45) What makes a good proxy task?

(06:09) Objective: has a reliable ground truth

(07:52) Nontrivial: existing methods dont crush it OOD (with some caveats)

(08:47) Tractable: the task is not impossible

(09:30) Confounder-free: methods cant cheat