The Nonlinear Library LW - On the Gladstone Report by Zvi
Mar 21, 2024
01:04:54
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On the Gladstone Report, published by Zvi on March 21, 2024 on LessWrong.
Like the the government-commissioned Gladstone Report on AI itself, there are two sections here.
First I cover the Gladstone Report's claims and arguments about the state of play, including what they learned talking to people inside the labs. I mostly agree with their picture and conclusions, both in terms of arguments and reported findings, however I already mostly agreed. If these arguments and this information is new to someone, and the form of a government-backed report helps them process it and take it seriously, this is good work.
However, in terms of convincing an already informed skeptic, I believe this is a failure. They did not present their findings in a way that should be found convincing to the otherwise unconvinced.
Second I cover the Gladstone Report's recommended courses of action. It is commendable that the report lays out a concrete, specific and highly detailed proposal. A lot of the details, and the broader outline, seem good. The compute thresholds seem too aggressive. I would suggest working to get agreement on the structure of intervention, while also talking price on the thresholds, and hopefully getting them to a better place.
Executive Summary of Their Findings: Oh No
According to the report, things are very much Not Great, Bob.
Here is their Twitter summary thread.
Edouard Harris: Here's what we've been working on for over a year:
The first US government-commissioned assessment of catastrophic national security risks from AI - including systems on the path to AGI.
TLDR: Things are worse than we thought. And nobody's in control.
We started this work with concerns, but no preconceptions. We knew there were solid technical reasons that AI could eventually pose catastrophic risks.
But we went in looking for reasons to change our minds.
We found the opposite.
Our overriding goal was to get to the truth. To do that, we had to do more than just speak to policy and leadership at the AI labs.
We also connected with individual technical researchers, many of whom are way more concerned than their labs let on in public.
Many of these folks came forward on condition of anonymity to share stories.
Let me tell you some of the most insane stuff we learned.
First off, inside one lab there's apparently a running joke that their security is so bad that they're doing more to accelerate the AI capabilities of US adversaries, than the adversaries themselves are.
Truly crazy. But this is where we're at.
It's a running joke, and also probably true, as I keep noticing. All our talk of 'but what about China' has to contend with the fact that China gets almost all its AI abilities directly from the United States. Some of it is spying. Some of it is training using our models. Some of it is seeing what is possible. Some of it is flat out open source and open model weights. But it is all on us.
Needless to say, if a major lab has this kind of running joke, that is completely unacceptable, everyone involved should be at least highly ashamed of themselves. More importantly, fix it.
More detail on this issue can be found in this good Time article, Employees at Top Labs Fear Safety Is an Afterthought, Report Says.
Quotes there are not reassuring.
In December we quietly polled a handful of frontier AI researchers and asked them:
What's the chance we end up on a path to a catastrophic AI outcome, *during the year 2024?*
We expected <1%. But no:
Lowest we got was 4%. Highest: up to 20%.
That's a wake-up call.
Catastrophic is very different from existential. If people were saying 4%-20% existential risk for 2024 alone, I would say those numbers are crazy high and make no sense. But this is a 4%-20% risk of a merely catastrophic outcome. If this is defined the way Anthropic defines it (so 1000+ deaths or 2...
Like the the government-commissioned Gladstone Report on AI itself, there are two sections here.
First I cover the Gladstone Report's claims and arguments about the state of play, including what they learned talking to people inside the labs. I mostly agree with their picture and conclusions, both in terms of arguments and reported findings, however I already mostly agreed. If these arguments and this information is new to someone, and the form of a government-backed report helps them process it and take it seriously, this is good work.
However, in terms of convincing an already informed skeptic, I believe this is a failure. They did not present their findings in a way that should be found convincing to the otherwise unconvinced.
Second I cover the Gladstone Report's recommended courses of action. It is commendable that the report lays out a concrete, specific and highly detailed proposal. A lot of the details, and the broader outline, seem good. The compute thresholds seem too aggressive. I would suggest working to get agreement on the structure of intervention, while also talking price on the thresholds, and hopefully getting them to a better place.
Executive Summary of Their Findings: Oh No
According to the report, things are very much Not Great, Bob.
Here is their Twitter summary thread.
Edouard Harris: Here's what we've been working on for over a year:
The first US government-commissioned assessment of catastrophic national security risks from AI - including systems on the path to AGI.
TLDR: Things are worse than we thought. And nobody's in control.
We started this work with concerns, but no preconceptions. We knew there were solid technical reasons that AI could eventually pose catastrophic risks.
But we went in looking for reasons to change our minds.
We found the opposite.
Our overriding goal was to get to the truth. To do that, we had to do more than just speak to policy and leadership at the AI labs.
We also connected with individual technical researchers, many of whom are way more concerned than their labs let on in public.
Many of these folks came forward on condition of anonymity to share stories.
Let me tell you some of the most insane stuff we learned.
First off, inside one lab there's apparently a running joke that their security is so bad that they're doing more to accelerate the AI capabilities of US adversaries, than the adversaries themselves are.
Truly crazy. But this is where we're at.
It's a running joke, and also probably true, as I keep noticing. All our talk of 'but what about China' has to contend with the fact that China gets almost all its AI abilities directly from the United States. Some of it is spying. Some of it is training using our models. Some of it is seeing what is possible. Some of it is flat out open source and open model weights. But it is all on us.
Needless to say, if a major lab has this kind of running joke, that is completely unacceptable, everyone involved should be at least highly ashamed of themselves. More importantly, fix it.
More detail on this issue can be found in this good Time article, Employees at Top Labs Fear Safety Is an Afterthought, Report Says.
Quotes there are not reassuring.
In December we quietly polled a handful of frontier AI researchers and asked them:
What's the chance we end up on a path to a catastrophic AI outcome, *during the year 2024?*
We expected <1%. But no:
Lowest we got was 4%. Highest: up to 20%.
That's a wake-up call.
Catastrophic is very different from existential. If people were saying 4%-20% existential risk for 2024 alone, I would say those numbers are crazy high and make no sense. But this is a 4%-20% risk of a merely catastrophic outcome. If this is defined the way Anthropic defines it (so 1000+ deaths or 2...