The Nonlinear Library LW - Stop talking about p(doom) by Isaac King
Jan 2, 2024
05:48
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Stop talking about p(doom), published by Isaac King on January 2, 2024 on LessWrong.
Epistemic status: Complete speculation, somewhat informed by copious arguing about the subject on Twitter.
As AI risk has moved into the mainstream over the past few years, I've come to believe that "p(doom)" is an actively harmful term for X-risk discourse, and people trying to mitigate X-risk should stop using it entirely.
Ambiguity
The first problem is that it's unclear what is actually being discussed. "p(doom)" can refer to many different things:
p(AI kills us within 5-10 years)
p(AI kills us within 80-200 years)
p(conditional on AGI, we die shortly afterwards)
p(conditional on superintelligence, we die shortly afterwards)
Like 10 other things.[1]
These could have wildly different probabilities, and come along with different cruxes for disagreement. Depending on what specific "doom" is being discussed, the relevant point could be any of:
Whether LLMs are capable of AGI at all.
Whether AGI will quickly turn into superintelligence.
Whether aligning superintelligence will be hard.
These are completely different questions, and people who are not explicit about which one they're discussing can end up talking past each other.
There are also many other potential miscommunications regarding exactly what "doom" refers to, the difference between one's inside view probability vs. ultimate probability, and more.
Distilling complex concepts down to single terms is good, but only when everyone is on the same page about what the term actually means.
Rhetoric
People concerned about X-risk tend to avoid "dark arts" rhetorical tactics, and justifiably so. Unfortunately, current society does not allow for complete good faith agents to do very well. Being fully honest about everything will turn you into a pariah, most people will judge you more based on charisma than on factual accuracy, and you need to use the right tribal signals before people will listen to you on a controversial topic at all. Using at least some light greyish arts in day to day life is necessary in order to succeed.
"p(doom)" is an extremely ineffective rhetorical tactic.
Motivated innumeracy
One of the most common responses from the e/acc crowd to discussions of p(doom) is to say that it's a made up, meaningless number, ungrounded in reality and therefore easily dismissed. Attempts to explain probability theory to them often end up with them denying the validity of probability theory entirely.
These sorts of motivated misunderstandings are extremely common, coming even from top physicists who suddenly lose their ability to understand high school level physics. Pointing out the isolated demand for rigor involved in their presumable acceptance of more pedestrian probabilistic statements also doesn't work; 60% of the time they ignore you entirely, the other 40% they retreat to extremely selective implementations of frequentistism where they're coincidentally able to define a base rate for any event that they have a probabilistic intuition for, and reject all other base rates as too speculative.
I think the fundamental issue here is that explicit probabilities are just weird to most people, and when they're being used to push a claim that is also weird, it's easy to see them as linked and reject everything coming from those people.
Framing AI risk in terms of Bayesian probability seems like a strategical error. People managed to convince the world of the dangers of climate change, nuclear war, asteroid impacts, and many other not-yet-clearly-demonstrated risks all without dying on the hill of Bayesian probability. They did of course make many probabilistic estimates, but restricted them to academic settings, and didn't frame the discussion largely in terms of specific numbers.
Normalizing the use of explicit probabilities is good,...
Epistemic status: Complete speculation, somewhat informed by copious arguing about the subject on Twitter.
As AI risk has moved into the mainstream over the past few years, I've come to believe that "p(doom)" is an actively harmful term for X-risk discourse, and people trying to mitigate X-risk should stop using it entirely.
Ambiguity
The first problem is that it's unclear what is actually being discussed. "p(doom)" can refer to many different things:
p(AI kills us within 5-10 years)
p(AI kills us within 80-200 years)
p(conditional on AGI, we die shortly afterwards)
p(conditional on superintelligence, we die shortly afterwards)
Like 10 other things.[1]
These could have wildly different probabilities, and come along with different cruxes for disagreement. Depending on what specific "doom" is being discussed, the relevant point could be any of:
Whether LLMs are capable of AGI at all.
Whether AGI will quickly turn into superintelligence.
Whether aligning superintelligence will be hard.
These are completely different questions, and people who are not explicit about which one they're discussing can end up talking past each other.
There are also many other potential miscommunications regarding exactly what "doom" refers to, the difference between one's inside view probability vs. ultimate probability, and more.
Distilling complex concepts down to single terms is good, but only when everyone is on the same page about what the term actually means.
Rhetoric
People concerned about X-risk tend to avoid "dark arts" rhetorical tactics, and justifiably so. Unfortunately, current society does not allow for complete good faith agents to do very well. Being fully honest about everything will turn you into a pariah, most people will judge you more based on charisma than on factual accuracy, and you need to use the right tribal signals before people will listen to you on a controversial topic at all. Using at least some light greyish arts in day to day life is necessary in order to succeed.
"p(doom)" is an extremely ineffective rhetorical tactic.
Motivated innumeracy
One of the most common responses from the e/acc crowd to discussions of p(doom) is to say that it's a made up, meaningless number, ungrounded in reality and therefore easily dismissed. Attempts to explain probability theory to them often end up with them denying the validity of probability theory entirely.
These sorts of motivated misunderstandings are extremely common, coming even from top physicists who suddenly lose their ability to understand high school level physics. Pointing out the isolated demand for rigor involved in their presumable acceptance of more pedestrian probabilistic statements also doesn't work; 60% of the time they ignore you entirely, the other 40% they retreat to extremely selective implementations of frequentistism where they're coincidentally able to define a base rate for any event that they have a probabilistic intuition for, and reject all other base rates as too speculative.
I think the fundamental issue here is that explicit probabilities are just weird to most people, and when they're being used to push a claim that is also weird, it's easy to see them as linked and reject everything coming from those people.
Framing AI risk in terms of Bayesian probability seems like a strategical error. People managed to convince the world of the dangers of climate change, nuclear war, asteroid impacts, and many other not-yet-clearly-demonstrated risks all without dying on the hill of Bayesian probability. They did of course make many probabilistic estimates, but restricted them to academic settings, and didn't frame the discussion largely in terms of specific numbers.
Normalizing the use of explicit probabilities is good,...