Accidental or Intentional Creation of Scary Optimizers?

Dr. Quintin Pope, PhD, is one of the few critics of AI doomerism who is truly fluent in the concepts and arguments. In Oct, 2023 he joined me for a debate in Twitter Spaces where he argued that AI alignment was basically already solved.

His “inside view” on machine-learning forced me to update my position, but could he knock me off the doom train?

Timestamps

00:00:00 — Cold Open

00:00:43 — Introductions

00:01:22 — Quintin's Opening Statement

00:02:32 — Liron's Opening Statement

00:05:10 — Has RLHF Solved the Alignment Problem?

00:07:52 — AI Capabilities Are Constrained by Training Data

00:10:52 — Defining ASI and Could RLHF Align a Superintelligence?

00:13:13 — Quintin Is More Optimistic Than OpenAI

00:14:16 — What Is ASI in Your Mind?

00:15:57 — AI in 5 Years (2028) & AI Coding Agents

00:19:05 — Continuous or Discontinuous Capability Gains?

00:19:39 — DEBATE: General Intelligence Algorithm in Humans

00:30:02 — The Only Coherent Explanation of Humans Going to the Moon

00:34:01 — Are We "Fully Cooked" as a General Optimizer?

00:35:53 — Common Mistake in Forecasting Superintelligence

00:42:22 — 'Neat' vs 'Scruffy': Will Interpretable Structure Emerge Inside Neural Nets?

00:48:57 — Does This Disagreement Actually Matter for P(Doom)?

00:54:33 — Thought Experiment: Could You Have Predicted a Species Would Go to the Moon?

00:57:26 — The Basin of Attraction for Superintelligence

00:59:35 — Does a Superintelligence Even Exist in Algorithm Space?

01:09:59 — Closing Statements

01:12:40 — Audience Q&A

01:19:35 — Wrap Up

Links

Original Twitter Spaces debate (Quintin Pope vs. Liron Shapira) — https://x.com/i/spaces/1YpJkwOzOqEJj/peek

Quintin Pope on Twitter/X — https://twitter.com/QuintinPope5

Quintin Pope, Alignment Forum profile — https://www.alignmentforum.org/users/quintin-pope

InstructGPT, Wikipedia — https://en.wikipedia.org/wiki/InstructGPT

AIXI, Wikipedia — https://en.wikipedia.org/wiki/AIXI

AlphaZero, Wikipedia — https://en.wikipedia.org/wiki/AlphaZero

MuZero, Wikipedia — https://en.wikipedia.org/wiki/MuZero

DeepMind AlphaZero and MuZero page — https://deepmind.google/research/alphazero-and-muzero/

Midjourney — https://www.midjourney.com/

DALL-E, Wikipedia — https://en.wikipedia.org/wiki/DALL-E

OpenAI Superalignment announcement — https://openai.com/index/introducing-superalignment/

Shard Theory sequence on LessWrong — https://www.lesswrong.com/s/nyEFg3AuJpdAozmoX

“Evolution Provides No Evidence for the Sharp Left Turn” — https://www.lesswrong.com/posts/hvz9qjWyv8cLX9JJR/evolution-provides-no-evidence-for-the-sharp-left-turn

“My Objections to ‘We’re All Gonna Die with Eliezer Yudkowsky’” — https://www.lesswrong.com/posts/wAczufCpMdaamF9fy/my-objections-to-we-re-all-gonna-die-with-eliezer-yudkowsky

“AI is Centralizing by Default; Let’s Not Make It Worse” — https://forum.effectivealtruism.org/posts/zd5inbT4kYKivincm/ai-is-centralizing-by-default-let-s-not-make-it-worse

Singular Learning Theory, Alignment Forum sequence — https://www.alignmentforum.org/s/mqwA5FcL6SrHEQzox

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏