The Nonlinear Library EA - Results from an Adversarial Collaboration on AI Risk (FRI) by Forecasting Research Institute
Mar 11, 2024
21:38
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Results from an Adversarial Collaboration on AI Risk (FRI), published by Forecasting Research Institute on March 11, 2024 on The Effective Altruism Forum.
Authors of linked report: Josh Rosenberg, Ezra Karger, Avital Morris, Molly Hickman, Rose Hadshar, Zachary Jacobs, Philip Tetlock[1]
Today, the Forecasting Research Institute (FRI) released "Roots of Disagreement on AI Risk: Exploring the Potential and Pitfalls of Adversarial Collaboration," which discusses the results of an adversarial collaboration focused on forecasting risks from AI.
In this post, we provide a brief overview of the methods, findings, and directions for further research. For much more analysis and discussion, see the full report: https://forecastingresearch.org/s/AIcollaboration.pdf
Abstract
We brought together generalist forecasters and domain experts (n=22) who disagreed about the risk AI poses to humanity in the next century. The "concerned" participants (all of whom were domain experts) predicted a 20% chance of an AI-caused existential catastrophe by 2100, while the "skeptical" group (mainly "superforecasters") predicted a 0.12% chance.
Participants worked together to find the strongest near-term cruxes: forecasting questions resolving by 2030 that would lead to the largest change in their beliefs (in expectation) about the risk of existential catastrophe by 2100.
Neither the concerned nor the skeptics substantially updated toward the other's views during our study, though one of the top short-term cruxes we identified is expected to close the gap in beliefs about AI existential catastrophe by about 5%: approximately 1 percentage point out of the roughly 20 percentage point gap in existential catastrophe forecasts.
We find greater agreement about a broader set of risks from AI over the next thousand years: the two groups gave median forecasts of 30% (skeptics) and 40% (concerned) that AI will have severe negative effects on humanity by causing major declines in population, very low self-reported well-being, or extinction.
Extended Executive Summary
In July 2023, we released our Existential Risk Persuasion Tournament (XPT) report, which identified large disagreements between domain experts and generalist forecasters about key risks to humanity (Karger et al. 2023). This new project - a structured adversarial collaboration run in April and May 2023 - is a follow-up to the XPT focused on better understanding the drivers of disagreement about AI risk.
Methods
We recruited participants to join "AI skeptic" (n=11) and "AI concerned" (n=11) groups that disagree strongly about the probability that AI will cause an existential catastrophe by 2100.[2] The skeptic group included nine superforecasters and two domain experts. The concerned group consisted of domain experts referred to us by staff members at Open Philanthropy (the funder of this project) and the broader Effective Altruism community.
Participants spent 8 weeks (skeptic median: 80 hours of work on the project; concerned median: 31 hours) reading background materials, developing forecasts, and engaging in online discussion and video calls.
We asked participants to work toward a better understanding of their sources of agreement and disagreement, and to propose and investigate "cruxes": short-term indicators, usually resolving by 2030, that would cause the largest updates in expectation to each group's view on the probability of existential catastrophe due to AI by 2100.
Results: What drives (and doesn't drive) disagreement over AI risk
At the beginning of the project, the median "skeptic" forecasted a 0.10% chance of existential catastrophe due to AI by 2100, and the median "concerned" participant forecasted a 25% chance. By the end, these numbers were 0.12% and 20% respectively, though many participants did not attribute their updates to a...
Authors of linked report: Josh Rosenberg, Ezra Karger, Avital Morris, Molly Hickman, Rose Hadshar, Zachary Jacobs, Philip Tetlock[1]
Today, the Forecasting Research Institute (FRI) released "Roots of Disagreement on AI Risk: Exploring the Potential and Pitfalls of Adversarial Collaboration," which discusses the results of an adversarial collaboration focused on forecasting risks from AI.
In this post, we provide a brief overview of the methods, findings, and directions for further research. For much more analysis and discussion, see the full report: https://forecastingresearch.org/s/AIcollaboration.pdf
Abstract
We brought together generalist forecasters and domain experts (n=22) who disagreed about the risk AI poses to humanity in the next century. The "concerned" participants (all of whom were domain experts) predicted a 20% chance of an AI-caused existential catastrophe by 2100, while the "skeptical" group (mainly "superforecasters") predicted a 0.12% chance.
Participants worked together to find the strongest near-term cruxes: forecasting questions resolving by 2030 that would lead to the largest change in their beliefs (in expectation) about the risk of existential catastrophe by 2100.
Neither the concerned nor the skeptics substantially updated toward the other's views during our study, though one of the top short-term cruxes we identified is expected to close the gap in beliefs about AI existential catastrophe by about 5%: approximately 1 percentage point out of the roughly 20 percentage point gap in existential catastrophe forecasts.
We find greater agreement about a broader set of risks from AI over the next thousand years: the two groups gave median forecasts of 30% (skeptics) and 40% (concerned) that AI will have severe negative effects on humanity by causing major declines in population, very low self-reported well-being, or extinction.
Extended Executive Summary
In July 2023, we released our Existential Risk Persuasion Tournament (XPT) report, which identified large disagreements between domain experts and generalist forecasters about key risks to humanity (Karger et al. 2023). This new project - a structured adversarial collaboration run in April and May 2023 - is a follow-up to the XPT focused on better understanding the drivers of disagreement about AI risk.
Methods
We recruited participants to join "AI skeptic" (n=11) and "AI concerned" (n=11) groups that disagree strongly about the probability that AI will cause an existential catastrophe by 2100.[2] The skeptic group included nine superforecasters and two domain experts. The concerned group consisted of domain experts referred to us by staff members at Open Philanthropy (the funder of this project) and the broader Effective Altruism community.
Participants spent 8 weeks (skeptic median: 80 hours of work on the project; concerned median: 31 hours) reading background materials, developing forecasts, and engaging in online discussion and video calls.
We asked participants to work toward a better understanding of their sources of agreement and disagreement, and to propose and investigate "cruxes": short-term indicators, usually resolving by 2030, that would cause the largest updates in expectation to each group's view on the probability of existential catastrophe due to AI by 2100.
Results: What drives (and doesn't drive) disagreement over AI risk
At the beginning of the project, the median "skeptic" forecasted a 0.10% chance of existential catastrophe due to AI by 2100, and the median "concerned" participant forecasted a 25% chance. By the end, these numbers were 0.12% and 20% respectively, though many participants did not attribute their updates to a...