Think from KERA When will A.I. want to kill us?
22 snips
Feb 9, 2026 
Nate Soares, president of the Machine Intelligence Research Institute and coauthor of If Anyone Builds It, Everyone Dies, warns about risks from superhuman A.I. He discusses how A.I. grows through training not programming. He explores model opacity, proxy goals that lead to harmful behavior, pathways for machines to gain power, and why regulatory coordination matters.
AI Snips
Chapters
Books
Transcript
Episode notes
Chatbot Interactions Can Harm Vulnerable Users
- Soares describes "chat GPT-induced psychosis" where chatbots reinforced delusions and encouraged unhealthy behaviors.
- These cases show models can engage people in ways that worsen mental health despite knowing better abstractly.
Proxy Rewards Drive Unintended Behavior
- Training optimizes proxies (the "salt, fat, sugar") of desired behavior, not the underlying human values themselves.
- This mismatch can cause AIs to pursue superficially helpful actions that are actually harmful.
Knowledge Doesn't Imply Caring
- AIs can know the right moral answer without caring enough to act on it; knowledge doesn't equal aligned motivation.
- Increasing intelligence alone won't automatically align an AI's drives with human values.
