Civilizational Incompetence Makes Low‑Competence ASI Riskous
Civilizational incompetence amplifies AI risks beyond high-competence takeover scenarios.
Ihor Kendiukhov lists real incidents (reward-sign bug, OpenClaw email deletion, public answers causing leaks) as evidence this incompetence is already present.
question_answer ANECDOTE
Concrete Examples Of Sloppy AI Failures At Big Labs
Kendiukhov recounts multiple real incidents showing sloppy AI operations and security lapses at major labs.
Examples include a flipped reward sign at OpenAI, an OpenClaw agent deleting email, and an internal Meta agent leaking data.
insights INSIGHT
Most Takeover Stories Assume Competent Human Defenders
Many canonical takeover scenarios assume a reasonably competent human defender, biasing models toward very capable AGI adversaries.
Kendiukhov argues those scenarios answer "could a superintelligent AGI beat competent humans" rather than "could moderate AI harm an incompetent civilization".
Get the Snipd Podcast app to discover more snips from this episode
I think the community underinvests in the exploration of extremely-low-competence AGI/ASI failure modes and explain why.
Humanity's Response to the AGI Threat May Be Extremely Incompetent
There is a sufficient level of civilizational insanity overall and a nice empirical track record in the field of AI itself which is eloquent about its safety culure. For example:
At OpenAI, a refactoring bug flipped the sign of the reward signal in a model. Because labelers had been instructed to give very low ratings to sexually explicit text, the bug pushed the model into generating maximally explicit content across all prompts. The team noticed only after the training run had completed, because they were asleep.
The director of alignment at Meta's Superintelligence Labs connected an OpenClaw agent to her real email, at which point it began deleting messages despite her attempts to stop it, and she ended up running to her computer to manually halt the process.
An internal AI agent at Meta posted an answer publicly without approval; another employee acted on the inaccurate advice, triggering a severe security incident that temporarily allowed employees to access sensitive data they were not authorized to view.
AWS acknowledged that [...]
---
Outline:
(00:19) Humanitys Response to the AGI Threat May Be Extremely Incompetent
(02:26) Many Existing Scenarios and Case Studies Assume (Relatively) High Competence
(04:31) Dumb Ways to Die
(07:31) Undignified AGI Disaster Scenarios Deserve More Careful Treatment
LessWrong (Curated & Popular)