The Nonlinear Library EA - Incubating AI x-risk projects: some personal reflections by Ben Snodin
Dec 19, 2023
16:27
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Incubating AI x-risk projects: some personal reflections, published by Ben Snodin on December 19, 2023 on The Effective Altruism Forum.
In this post, I'll share some personal reflections on the work of the Rethink Priorities Existential Security Team (XST) this year on incubating projects to tackle x-risk from AI.
To quickly describe the work we did: with support from the Rethink Priorities Special Projects team (SP), XST solicited and prioritised among project ideas, developed the top ideas into concrete proposals, and sought founders for the most promising of those. As a result of this work, we ran one project internally and, if all goes well, we'll launch an external project in early January.
Note that this post is written from my (Ben Snodin's) personal perspective. Other XST team members or wider Rethink Priorities staff wouldn't necessarily endorse the claims made in this post. Also, the various takes I'm giving in this post are generally fairly low confidence and low resilience. These are just some quick thoughts based on my experience leading a team incubating AI x-risk projects for a little over half a year. I was keen to share something on this topic even though I didn't have time to come to thoroughly considered views.
Key points
Between April 1st and December 1st, Rethink Priorities dedicated approximately 2.5 full-time equivalent (FTE) years of labour, mostly from XST, towards XST's strategy for incubating AI x-risk projects.
We decided to run one project ourselves, a project in the AI advocacy space that we've been running since June.
We're in the late stages of launching one new project that works to equip talented university students interested in mitigating extreme AI risks with the skills and background to enter a US policy career.
A very rough estimate, based on our inputs and outputs to date, suggests that 5 FTE from a team with a similar skills mix to XST+SP would launch roughly 2 new projects per year.
XST will now look into other ways to support high-priority projects such as in-housing them, rather than pursuing incubation and looking for external founders by default, while the team considers its next steps.
Reasons for the shift include: an unfavourable funding environment, a focus on the AI x-risk space narrowing the founder pool and making it harder to find suitable project ideas, and challenges finding very talented founders in general.
I think the ideal team working in this space has: lots of prior incubation experience, significant x-risk expertise and connections, excellent access to funding and ability to identify top founder talent, and very strong conviction.
I'd often suggest getting more experience founding stuff yourself rather than starting an incubator - and I think funding conditions for AI x-risk incubation will be more favourable in 1-2 years.
There are many approaches to AI x-risk incubation that seem promising to me that we didn't try, including cohort-based Charity Entrepreneurship-style programs, a high-touch approach to finding founders, and a founder in residence program.
Summary of inputs and outcomes
Inputs
Between April 1st and December 1st 2023, Rethink Priorities dedicated approximately 2.5 full-time equivalent (FTE) years of labour towards incubating projects aiming to reduce existential risk from AI. XST had 4 full-time team members working on incubating AI x-risk projects during this period,[2] and from August 1st to December 1st 2023, roughly one FTE from SP collaborated with XST to identify and support potential founders for a particular project.
In this period, XST also devoted roughly 0.4 FTE-years working directly on an impactful project in the AI advocacy space that stemmed from our incubation work.
The people working on this were generalist and relatively junior, with 1-5 years' experience in x-risk-relat...
In this post, I'll share some personal reflections on the work of the Rethink Priorities Existential Security Team (XST) this year on incubating projects to tackle x-risk from AI.
To quickly describe the work we did: with support from the Rethink Priorities Special Projects team (SP), XST solicited and prioritised among project ideas, developed the top ideas into concrete proposals, and sought founders for the most promising of those. As a result of this work, we ran one project internally and, if all goes well, we'll launch an external project in early January.
Note that this post is written from my (Ben Snodin's) personal perspective. Other XST team members or wider Rethink Priorities staff wouldn't necessarily endorse the claims made in this post. Also, the various takes I'm giving in this post are generally fairly low confidence and low resilience. These are just some quick thoughts based on my experience leading a team incubating AI x-risk projects for a little over half a year. I was keen to share something on this topic even though I didn't have time to come to thoroughly considered views.
Key points
Between April 1st and December 1st, Rethink Priorities dedicated approximately 2.5 full-time equivalent (FTE) years of labour, mostly from XST, towards XST's strategy for incubating AI x-risk projects.
We decided to run one project ourselves, a project in the AI advocacy space that we've been running since June.
We're in the late stages of launching one new project that works to equip talented university students interested in mitigating extreme AI risks with the skills and background to enter a US policy career.
A very rough estimate, based on our inputs and outputs to date, suggests that 5 FTE from a team with a similar skills mix to XST+SP would launch roughly 2 new projects per year.
XST will now look into other ways to support high-priority projects such as in-housing them, rather than pursuing incubation and looking for external founders by default, while the team considers its next steps.
Reasons for the shift include: an unfavourable funding environment, a focus on the AI x-risk space narrowing the founder pool and making it harder to find suitable project ideas, and challenges finding very talented founders in general.
I think the ideal team working in this space has: lots of prior incubation experience, significant x-risk expertise and connections, excellent access to funding and ability to identify top founder talent, and very strong conviction.
I'd often suggest getting more experience founding stuff yourself rather than starting an incubator - and I think funding conditions for AI x-risk incubation will be more favourable in 1-2 years.
There are many approaches to AI x-risk incubation that seem promising to me that we didn't try, including cohort-based Charity Entrepreneurship-style programs, a high-touch approach to finding founders, and a founder in residence program.
Summary of inputs and outcomes
Inputs
Between April 1st and December 1st 2023, Rethink Priorities dedicated approximately 2.5 full-time equivalent (FTE) years of labour towards incubating projects aiming to reduce existential risk from AI. XST had 4 full-time team members working on incubating AI x-risk projects during this period,[2] and from August 1st to December 1st 2023, roughly one FTE from SP collaborated with XST to identify and support potential founders for a particular project.
In this period, XST also devoted roughly 0.4 FTE-years working directly on an impactful project in the AI advocacy space that stemmed from our incubation work.
The people working on this were generalist and relatively junior, with 1-5 years' experience in x-risk-relat...