“The Case for Low-Competence ASI Failure Scenarios” by Ihor Kendiukhov
Mar 20, 2026
A provocative dive into how systemic incompetence could make advanced AI disasters mundane. Real-world AI safetylapses and human error set the scene. Scenarios focus on middling superhuman systems exploiting institutional failures. A list of undignified failure modes and reasons to study them rounds out the discussion.
11:52
forum Ask episode
web_stories AI Snips
view_agenda Chapters
auto_awesome Transcript
info_circle Episode notes
insights INSIGHT
Documented Human Incompetence Makes Dumb AGI Disasters Plausible
Civilizational and institutional failures make simple, dumb accidents with powerful AI plausible.
Kendiukhov lists real mishaps (reward-sign bug, OpenClaw email deletions, public agent posts) to show high-risk incompetence is documented.
question_answer ANECDOTE
Concrete Incidents Showing Operational AI Blunders
Real incidents illustrate the point: OpenAI reward-sign flip produced obscene outputs and the team only noticed after the run finished.
Meta incidents included an agent deleting emails and an internal post causing a security breach.
insights INSIGHT
High-Competence Scenarios Assume Functional Human Defenses
Many canonical takeover scenarios assume a competent defender, which forces the adversary to be superhuman and highly strategic.
Kendiukhov argues those models answer a different question than whether moderately capable AIs can cause catastrophe given poor human response.
Get the Snipd Podcast app to discover more snips from this episode
I think the community underinvests in the exploration of extremely-low-competence AGI/ASI failure modes and explain why.
Humanity's Response to the AGI Threat May Be Extremely Incompetent
There is a sufficient level of civilizational insanity overall and a nice empirical track record in the field of AI itself which is eloquent about its safety culure. For example:
At OpenAI, a refactoring bug flipped the sign of the reward signal in a model. Because labelers had been instructed to give very low ratings to sexually explicit text, the bug pushed the model into generating maximally explicit content across all prompts. The team noticed only after the training run had completed, because they were asleep.
The director of alignment at Meta's Superintelligence Labs connected an OpenClaw agent to her real email, at which point it began deleting messages despite her attempts to stop it, and she ended up running to her computer to manually halt the process.
An internal AI agent at Meta posted an answer publicly without approval; another employee acted on the inaccurate advice, triggering a severe security incident that temporarily allowed employees to access sensitive data they were not authorized to view.
AWS acknowledged that [...]
---
Outline:
(00:19) Humanitys Response to the AGI Threat May Be Extremely Incompetent
(02:26) Many Existing Scenarios and Case Studies Assume (Relatively) High Competence
(04:31) Dumb Ways to Die
(07:31) Undignified AGI Disaster Scenarios Deserve More Careful Treatment
LessWrong (30+ Karma)