"Schelling Goodness, and Shared Morality as a Goal" by Andrew_Critch
LessWrong (Curated & Popular)
00:00
Implications: instrumentality and why consider shelling norms
Andrew Critch lists instrumental reasons agents should consider cosmic shelling morality for coordination.
Play episode from 51:44
Transcript
Transcript
Episode notes
Also available in markdown at theMultiplicity.ai/blog/schelling-goodness.
This post explores a notion I'll call Schelling goodness. Claims of Schelling goodness are not first-order moral verdicts like "X is good" or "X is bad." They are claims about a class of hypothetical coordination games in the sense of Thomas Schelling, where the task being coordinated on is a moral verdict. In each such game, participants aim to give the same response regarding a moral question, by reasoning about what a very diverse population of intelligent beings would converge on, using only broadly shared constraints: common knowledge of the question at hand, and background knowledge from the survival and growth pressures that shape successful civilizations. Unlike many Schelling coordination games, we'll be focused on scenarios with no shared history or knowledge amongst the participants, other than being from successful civilizations.
Importantly: To say "X is Schelling-good" is not at all the same as saying "X is good". Rather, it will be defined as a claim about what a large class of agents would say, if they were required to choose between saying "X is good" and "X is bad" and aiming for a mutually agreed-upon answer. This distinction is crucial [...]
---
Outline:
(01:59) This essay is not very skimmable
(03:44) Pro tanto morals, is good, and is bad
(06:39) Part One: The Schelling Participation Effect
(13:52) What makes it work
(15:50) The Schelling transformation on questions
(19:10) Part Two: Schelling morality via the cosmic Schelling population
(21:12) Scale-invariant adaptations
(22:54) An example: stealing
(30:32) Recognition versus endorsement versus adherence
(31:34) The answer frequencies versus the answer
(33:59) Ties are rare
(35:06) Is the cosmic Schelling answer ever knowable with confidence?
(36:02) Schelling participation effects, revisited
(38:03) Is this just the mind projection fallacy?
(39:42) When are cosmic Schelling morals easy to identify?
(42:59) Scale invariance revisited
(44:03) A second example: Pareto-positive trade
(47:45) Harder questions and caveats
(50:01) Ties are unstable
(51:43) Isnt this assuming moral realism?
(53:07) Dont these results depend on the distribution over beings?
(54:41) What about the is-ought gap?
(56:29) Tolerance, local variation, and freedom
(58:25) Terrestrial Schelling-goodness
(59:42) So what does good mean, again?
(01:01:08) Implications for AI alignment
(01:06:15) Conclusion and historical context
(01:09:16) FAQ
(01:09:20) Basic misunderstandings
(01:12:20) More nuanced questions
---
First published:
February 28th, 2026
Source:
https://www.lesswrong.com/posts/TkBCR8XRGw7qmao6z/schelling-goodness-and-shared-morality-as-a-goal
---
Narrated by TYPE III AUDIO.
This post explores a notion I'll call Schelling goodness. Claims of Schelling goodness are not first-order moral verdicts like "X is good" or "X is bad." They are claims about a class of hypothetical coordination games in the sense of Thomas Schelling, where the task being coordinated on is a moral verdict. In each such game, participants aim to give the same response regarding a moral question, by reasoning about what a very diverse population of intelligent beings would converge on, using only broadly shared constraints: common knowledge of the question at hand, and background knowledge from the survival and growth pressures that shape successful civilizations. Unlike many Schelling coordination games, we'll be focused on scenarios with no shared history or knowledge amongst the participants, other than being from successful civilizations.
Importantly: To say "X is Schelling-good" is not at all the same as saying "X is good". Rather, it will be defined as a claim about what a large class of agents would say, if they were required to choose between saying "X is good" and "X is bad" and aiming for a mutually agreed-upon answer. This distinction is crucial [...]
---
Outline:
(01:59) This essay is not very skimmable
(03:44) Pro tanto morals, is good, and is bad
(06:39) Part One: The Schelling Participation Effect
(13:52) What makes it work
(15:50) The Schelling transformation on questions
(19:10) Part Two: Schelling morality via the cosmic Schelling population
(21:12) Scale-invariant adaptations
(22:54) An example: stealing
(30:32) Recognition versus endorsement versus adherence
(31:34) The answer frequencies versus the answer
(33:59) Ties are rare
(35:06) Is the cosmic Schelling answer ever knowable with confidence?
(36:02) Schelling participation effects, revisited
(38:03) Is this just the mind projection fallacy?
(39:42) When are cosmic Schelling morals easy to identify?
(42:59) Scale invariance revisited
(44:03) A second example: Pareto-positive trade
(47:45) Harder questions and caveats
(50:01) Ties are unstable
(51:43) Isnt this assuming moral realism?
(53:07) Dont these results depend on the distribution over beings?
(54:41) What about the is-ought gap?
(56:29) Tolerance, local variation, and freedom
(58:25) Terrestrial Schelling-goodness
(59:42) So what does good mean, again?
(01:01:08) Implications for AI alignment
(01:06:15) Conclusion and historical context
(01:09:16) FAQ
(01:09:20) Basic misunderstandings
(01:12:20) More nuanced questions
---
First published:
February 28th, 2026
Source:
https://www.lesswrong.com/posts/TkBCR8XRGw7qmao6z/schelling-goodness-and-shared-morality-as-a-goal
---
Narrated by TYPE III AUDIO.
The AI-powered Podcast Player
Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
