The Nonlinear Library LW - MIRI 2024 Mission and Strategy Update by Malo
Jan 5, 2024
16:08
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: MIRI 2024 Mission and Strategy Update, published by Malo on January 5, 2024 on LessWrong.
As we
announced back in October, I have taken on the senior leadership role at MIRI as its CEO. It's a big pair of shoes to fill, and an awesome responsibility that I'm honored to take on.
There have been several changes at MIRI since
our 2020 strategic update, so let's get into it.[1]
The short version:
We think it's very unlikely that the AI alignment field will be able to make progress quickly enough to prevent human extinction and the loss of the future's potential value, that we expect will result from loss of control to smarter-than-human AI systems.
However, developments this past year like the release of ChatGPT seem to have shifted the
Overton window in a lot of groups. There's been a lot more discussion of extinction risk from AI, including among policymakers, and the discussion quality seems greatly improved.
This provides a glimmer of hope. While we expect that more shifts in public opinion are necessary before the world takes actions that sufficiently change its course, it now appears more likely that governments could enact meaningful regulations to forestall the development of unaligned, smarter-than-human AI systems. It also seems more possible that humanity could take on a new megaproject squarely aimed at ending the acute risk period.
As such, in 2023, MIRI shifted its strategy to pursue three objectives:
Policy: Increase the probability that the major governments of the world end up coming to some international agreement to halt progress toward smarter-than-human AI, until humanity's state of knowledge and justified confidence about its understanding of relevant phenomena has drastically changed; and until we are able to secure these systems such that they can't fall into the hands of malicious or incautious actors.[2]
Communications: Share our models of the situation with a broad audience, especially in cases where talking about an important consideration could help normalize discussion of it.
Research: Continue to invest in a portfolio of research. This includes technical alignment research (though we've become more pessimistic that such work will have time to bear fruit if policy interventions fail to buy the research field more time), as well as research in support of our policy and communications goals.[3]
We see the communications work as instrumental support for our policy objective. We also see candid and honest communication as a way to bring key models and considerations into the Overton window, and we generally think that being honest in this way tends to be a good default.
Although we plan to pursue all three of these priorities, it's likely that policy and communications will be a higher priority for MIRI than research going forward.[4]
The rest of this post will discuss MIRI's trajectory over time and our current strategy. In one or more future posts, we plan to say more about our policy/comms efforts and our research plans.
Note that this post will assume that you're already reasonably familiar with MIRI and AGI risk; if you aren't, I recommend checking out Eliezer Yudkowsky's recent short TED talk,
along with some of the resources cited on the
TED page:
"
A.I. Poses 'Risk of Extinction,' Industry Leaders Warn", New York Times
"
We must slow down the race to god-like AI", Financial Times
"
Pausing AI Developments Isn't Enough. We Need to Shut it All Down", TIME
"
AGI Ruin: A List of Lethalities", AI Alignment Forum
MIRI's mission
Throughout its history, MIRI's goal has been to ensure that the long-term future goes well, with a focus on increasing the probability that humanity can safely navigate the transition to a world with smarter-than-human AI. If humanity can safely navigate the emergence of these systems, we believe this will lead to unpre...
As we
announced back in October, I have taken on the senior leadership role at MIRI as its CEO. It's a big pair of shoes to fill, and an awesome responsibility that I'm honored to take on.
There have been several changes at MIRI since
our 2020 strategic update, so let's get into it.[1]
The short version:
We think it's very unlikely that the AI alignment field will be able to make progress quickly enough to prevent human extinction and the loss of the future's potential value, that we expect will result from loss of control to smarter-than-human AI systems.
However, developments this past year like the release of ChatGPT seem to have shifted the
Overton window in a lot of groups. There's been a lot more discussion of extinction risk from AI, including among policymakers, and the discussion quality seems greatly improved.
This provides a glimmer of hope. While we expect that more shifts in public opinion are necessary before the world takes actions that sufficiently change its course, it now appears more likely that governments could enact meaningful regulations to forestall the development of unaligned, smarter-than-human AI systems. It also seems more possible that humanity could take on a new megaproject squarely aimed at ending the acute risk period.
As such, in 2023, MIRI shifted its strategy to pursue three objectives:
Policy: Increase the probability that the major governments of the world end up coming to some international agreement to halt progress toward smarter-than-human AI, until humanity's state of knowledge and justified confidence about its understanding of relevant phenomena has drastically changed; and until we are able to secure these systems such that they can't fall into the hands of malicious or incautious actors.[2]
Communications: Share our models of the situation with a broad audience, especially in cases where talking about an important consideration could help normalize discussion of it.
Research: Continue to invest in a portfolio of research. This includes technical alignment research (though we've become more pessimistic that such work will have time to bear fruit if policy interventions fail to buy the research field more time), as well as research in support of our policy and communications goals.[3]
We see the communications work as instrumental support for our policy objective. We also see candid and honest communication as a way to bring key models and considerations into the Overton window, and we generally think that being honest in this way tends to be a good default.
Although we plan to pursue all three of these priorities, it's likely that policy and communications will be a higher priority for MIRI than research going forward.[4]
The rest of this post will discuss MIRI's trajectory over time and our current strategy. In one or more future posts, we plan to say more about our policy/comms efforts and our research plans.
Note that this post will assume that you're already reasonably familiar with MIRI and AGI risk; if you aren't, I recommend checking out Eliezer Yudkowsky's recent short TED talk,
along with some of the resources cited on the
TED page:
"
A.I. Poses 'Risk of Extinction,' Industry Leaders Warn", New York Times
"
We must slow down the race to god-like AI", Financial Times
"
Pausing AI Developments Isn't Enough. We Need to Shut it All Down", TIME
"
AGI Ruin: A List of Lethalities", AI Alignment Forum
MIRI's mission
Throughout its history, MIRI's goal has been to ensure that the long-term future goes well, with a focus on increasing the probability that humanity can safely navigate the transition to a world with smarter-than-human AI. If humanity can safely navigate the emergence of these systems, we believe this will lead to unpre...