The Nonlinear Library AF - Auditing failures vs concentrated failures by Ryan Greenblatt
Dec 11, 2023
13:09
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Auditing failures vs concentrated failures, published by Ryan Greenblatt on December 11, 2023 on The AI Alignment Forum.
TL;DR: If you want to notice when powerful AIs are behaving badly, a reasonable strategy is to carefully audit a small fraction of AI actions. This kind of strategy has two kinds of failures which can be tackled somewhat independently: auditing failures, where the audits fail to recognize some bad behavior, and concentrated failures, where the AI causes an irreversible catastrophe in so few actions or such a short time frame that auditing didn't have a chance to review the bad actions.
Many oversight techniques are so expensive that using them on every AI action would destroy most of the value of using AIs in the first place: you might want to have humans extensively review all AI-generated code line by line, but this reduces AI usefulness so much that you would often be better off having humans write the code in the first place.
A typical way to solve this problem is to just audit actions during an upfront training period (either to directly train an AI, or to train a
reward model), but this fails if AIs can confidently tell some deployment inputs from training inputs: they could behave arbitrarily badly on these distinguishable deployment inputs by using a strategy like "if in training or uncertain do X; if in deployment do Y".
Here is a better baseline: random audits. In addition to training, apply expensive human auditing to a randomly selected set of deployment actions (e.g. 2% of actions) and check that AIs aren't doing more bad actions than expected. Now for AIs to frequently behave poorly without getting caught, AIs can't just depend on distinguishing training from some deployment inputs: AIs must be able to sneak bad behavior past the auditing process, otherwise, auditing would eventually notice bad actions[1].
Random audits don't solve all issues, there are still:
Auditing failures: The auditing process failed to recognize bad behavior[2] (this is the classic scalable oversight problem).
Concentrated failures: We failed to handle catastrophic behavior because the catastrophic behavior is concentrated in a small number of actions or a small time frame (these are sometimes called high-stakes failures[3]). This can be further decomposed into two failure modes:
Sampling concentrated failure: The auditing process didn't sample the concentrated catastrophic actions. This is catastrophic if AIs can cause a catastrophe in a small number of actions. (For instance, if AIs can cause a catastrophe in 20 bad actions and we only audit 2% of actions, then we would only sample at least one bad action with probability 0.33.).
Latency concentrated failure: The auditing process failed to finish reviewing the catastrophic actions before it was too late. This is catastrophic if AIs can coordinate to take enough bad actions in a small amount of time to irreversibly take over.
We believe this distinction is often useful when thinking about what different techniques are useful for: sometimes they help you make the expensive auditing better, sometimes they help you audit more the actions that matter the most, and sometimes they help you audit before it's too late.[4]
The rest of this post dives deeper into concentrated failures and describes some ways to improve on the random auditing baseline.
Sampling concentrated failure
AIs could take bad actions rarely, such that they fail a small number of times in expectation throughout deployment and are thus unlikely to behave badly in the small fraction of AI actions which are audited (as seen in our paper). We call this a sampling concentrated failure as the issue is that the AI took bad actions very rarely, which resulted in auditing not sampling bad actions.
Latency concentrated failure
AIs could coordinate on doing ...
TL;DR: If you want to notice when powerful AIs are behaving badly, a reasonable strategy is to carefully audit a small fraction of AI actions. This kind of strategy has two kinds of failures which can be tackled somewhat independently: auditing failures, where the audits fail to recognize some bad behavior, and concentrated failures, where the AI causes an irreversible catastrophe in so few actions or such a short time frame that auditing didn't have a chance to review the bad actions.
Many oversight techniques are so expensive that using them on every AI action would destroy most of the value of using AIs in the first place: you might want to have humans extensively review all AI-generated code line by line, but this reduces AI usefulness so much that you would often be better off having humans write the code in the first place.
A typical way to solve this problem is to just audit actions during an upfront training period (either to directly train an AI, or to train a
reward model), but this fails if AIs can confidently tell some deployment inputs from training inputs: they could behave arbitrarily badly on these distinguishable deployment inputs by using a strategy like "if in training or uncertain do X; if in deployment do Y".
Here is a better baseline: random audits. In addition to training, apply expensive human auditing to a randomly selected set of deployment actions (e.g. 2% of actions) and check that AIs aren't doing more bad actions than expected. Now for AIs to frequently behave poorly without getting caught, AIs can't just depend on distinguishing training from some deployment inputs: AIs must be able to sneak bad behavior past the auditing process, otherwise, auditing would eventually notice bad actions[1].
Random audits don't solve all issues, there are still:
Auditing failures: The auditing process failed to recognize bad behavior[2] (this is the classic scalable oversight problem).
Concentrated failures: We failed to handle catastrophic behavior because the catastrophic behavior is concentrated in a small number of actions or a small time frame (these are sometimes called high-stakes failures[3]). This can be further decomposed into two failure modes:
Sampling concentrated failure: The auditing process didn't sample the concentrated catastrophic actions. This is catastrophic if AIs can cause a catastrophe in a small number of actions. (For instance, if AIs can cause a catastrophe in 20 bad actions and we only audit 2% of actions, then we would only sample at least one bad action with probability 0.33.).
Latency concentrated failure: The auditing process failed to finish reviewing the catastrophic actions before it was too late. This is catastrophic if AIs can coordinate to take enough bad actions in a small amount of time to irreversibly take over.
We believe this distinction is often useful when thinking about what different techniques are useful for: sometimes they help you make the expensive auditing better, sometimes they help you audit more the actions that matter the most, and sometimes they help you audit before it's too late.[4]
The rest of this post dives deeper into concentrated failures and describes some ways to improve on the random auditing baseline.
Sampling concentrated failure
AIs could take bad actions rarely, such that they fail a small number of times in expectation throughout deployment and are thus unlikely to behave badly in the small fraction of AI actions which are audited (as seen in our paper). We call this a sampling concentrated failure as the issue is that the AI took bad actions very rarely, which resulted in auditing not sampling bad actions.
Latency concentrated failure
AIs could coordinate on doing ...