(Podcast version, read by the author, here, or search for "Joe Carlsmith Audio" on your podcast app.
This is the tenth essay in a series I’m calling “How do we solve the alignment problem?”. I’m hoping that the individual essays can be read fairly well on their own, but see this introduction for a summary of the essays that have been released thus far, plus a bit more about the series as a whole.
I work at Anthropic, but I am here speaking only for myself and not for my employer.)
1. Introduction
In the third essay in the series, I distinguished between three key “security factors” for developing advanced AI safely, namely:
- Safety progress: our ability to develop new levels of AI capability safely.
- Risk evaluation: our ability to track and forecast the level of risk that a given sort of AI capability development involves.
- Capability restraint: our ability to steer and restrain AI capability development when doing so is necessary for maintaining safety.
A lot of my focus in the series has been on safety progress – and to a lesser extent, risk evaluation. In this essay, I want to look at capability restraint, in [...]
---
Outline:
(00:38) 1. Introduction
(08:18) 2. Preliminaries
(10:59) 3. AI development isnt necessarily a prisoners dilemma
(18:03) 4. Forms of capability restraint
(19:26) 4.1. Individual capability restraint
(21:32) 4.2. Collective capability restraint
(26:25) 4.3. Treatment of ongoing AI development
(33:06) 5. Idealized capability restraint
(45:00) 6. Capability restraint in practice
(45:32) 6.1. The likelihood of serious effort
(53:57) 6.2. The efficacy of capability restraint
(55:40) 6.2.1. Compute governance
(58:05) 6.2.2. Algorithmic governance
(01:04:12) 6.2.3. Greenlighting and safety progress
(01:10:44) 6.3. Ways that capability restraint could end up net negative
(01:11:53) 6.3.1. Concentrations of power
(01:17:17) 6.3.2. Ceding competitive advantage to authoritarian countries
(01:19:29) 6.3.3. Other concerns
(01:27:13) 7. Prioritizing capability restraint relative to other security factors
(01:29:43) 8. Conclusion
(01:31:19) Appendix 1: What are we using the time for?
The original text contained 40 footnotes which were omitted from this narration.
---
First published:
March 19th, 2026
Source:
https://www.lesswrong.com/posts/K8jyKcDQbfBjmiAoM/on-restraining-ai-development-for-the-sake-of-safety
---
Narrated by TYPE III AUDIO.
