The Nonlinear Library EA - Good job opportunities for helping with the most important century by Holden Karnofsky
Jan 18, 2024
07:34
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Good job opportunities for helping with the most important century, published by Holden Karnofsky on January 18, 2024 on The Effective Altruism Forum.
Yes, this is my first post in almost a year. I'm no longer prioritizing this blog, but I will still occasionally post something.
I wrote ~2 years ago that it was hard to point to concrete opportunities to help the most important century go well. That's changing.
There are a good number of jobs available now that are both really promising opportunities to help (in my opinion) and are suitable for people without a lot of pre-existing knowledge of AI risk (or even AI). The jobs are demanding, but unlike many of the job openings that existed a couple of years ago, they are at well-developed organizations and involve relatively clear goals.
So if you're someone who wants to help, but has been waiting for the right moment, this might be it. (Or not! I'll probably keep making posts like this as the set of opportunities gets wider.)
Here are the jobs that best fit this description right now, as far as I can tell. The rest of this post will give a bit more detail on how these jobs can help, what skills they require and why these are the ones I listed.
Organization
Location
Jobs
Link
UK AI Safety Institute
London (remote work possible within the UK)
Engineering and frontend roles, cybersecurity roles
Here
AAAS, Horizon Institute for Public Service, Tech Congress
Washington, DC
Fellowships serving as entry points into US policy roles
Here
AI companies: Google DeepMind, OpenAI, Anthropic1
San Francisco and London (with some other offices and remote work options)
Preparedness/Responsible Scaling roles; alignment research roles
Here, here, here, here
Model Evaluation and Threat Research (METR) (fewer roles available)
Berkeley (with remote work options)
Engineering and data roles
Here
Software engineering and development (and related areas) seem especially valuable right now, so think about whether you know folks with those skills who might be interested!
How these help
A lot of these jobs (and the ones I know the most about) would be contributing toward a possible global standards regime for AI: AI systems should be subject to testing to see whether they present major risks, and training/deploying AI should stopped (e.g., by regulation) when it can't be done safely.
The basic hope is:
Teams will develop "evals": tests of what AIs are capable of, particularly with respect to possible risks. For example, one eval might be prompting an AI to give a detailed description of how to build a bioweapon; the more detailed and accurate its response, the more risk the AI poses (while also possibly having more potential benefits as well, by virtue of being generally more knowledgeable/capable).
It will become common (through regulation, voluntary action by companies, industry standards, etc.) for cutting-edge AI systems to be subject to evals for dangerous capabilities.
When evals reveal risk, they will trigger required mitigations. For example:
An AI capable of bioweapons development should be (a) deployed in such a way that people can't use it for that (including by "jailbreaking" it), and (b) kept under good security to stop would-be terrorists from circumventing the restrictions.
AIs with stronger and more dangerous capabilities might require very challenging mitigations, possibly beyond what anyone knows how to do today (for example, rigorous demonstrations that an AI won't have dangerous unintended aims, even if this sort of thing is hard to measure).
Ideally, we'd eventually build a robust international governance regime (comparisons have been made to nuclear non-proliferation regimes) that reliably enforces rules like these, while safe and beneficial AI goes forward. But my view is that even dramatically weaker setups can still help a lo...
Yes, this is my first post in almost a year. I'm no longer prioritizing this blog, but I will still occasionally post something.
I wrote ~2 years ago that it was hard to point to concrete opportunities to help the most important century go well. That's changing.
There are a good number of jobs available now that are both really promising opportunities to help (in my opinion) and are suitable for people without a lot of pre-existing knowledge of AI risk (or even AI). The jobs are demanding, but unlike many of the job openings that existed a couple of years ago, they are at well-developed organizations and involve relatively clear goals.
So if you're someone who wants to help, but has been waiting for the right moment, this might be it. (Or not! I'll probably keep making posts like this as the set of opportunities gets wider.)
Here are the jobs that best fit this description right now, as far as I can tell. The rest of this post will give a bit more detail on how these jobs can help, what skills they require and why these are the ones I listed.
Organization
Location
Jobs
Link
UK AI Safety Institute
London (remote work possible within the UK)
Engineering and frontend roles, cybersecurity roles
Here
AAAS, Horizon Institute for Public Service, Tech Congress
Washington, DC
Fellowships serving as entry points into US policy roles
Here
AI companies: Google DeepMind, OpenAI, Anthropic1
San Francisco and London (with some other offices and remote work options)
Preparedness/Responsible Scaling roles; alignment research roles
Here, here, here, here
Model Evaluation and Threat Research (METR) (fewer roles available)
Berkeley (with remote work options)
Engineering and data roles
Here
Software engineering and development (and related areas) seem especially valuable right now, so think about whether you know folks with those skills who might be interested!
How these help
A lot of these jobs (and the ones I know the most about) would be contributing toward a possible global standards regime for AI: AI systems should be subject to testing to see whether they present major risks, and training/deploying AI should stopped (e.g., by regulation) when it can't be done safely.
The basic hope is:
Teams will develop "evals": tests of what AIs are capable of, particularly with respect to possible risks. For example, one eval might be prompting an AI to give a detailed description of how to build a bioweapon; the more detailed and accurate its response, the more risk the AI poses (while also possibly having more potential benefits as well, by virtue of being generally more knowledgeable/capable).
It will become common (through regulation, voluntary action by companies, industry standards, etc.) for cutting-edge AI systems to be subject to evals for dangerous capabilities.
When evals reveal risk, they will trigger required mitigations. For example:
An AI capable of bioweapons development should be (a) deployed in such a way that people can't use it for that (including by "jailbreaking" it), and (b) kept under good security to stop would-be terrorists from circumventing the restrictions.
AIs with stronger and more dangerous capabilities might require very challenging mitigations, possibly beyond what anyone knows how to do today (for example, rigorous demonstrations that an AI won't have dangerous unintended aims, even if this sort of thing is hard to measure).
Ideally, we'd eventually build a robust international governance regime (comparisons have been made to nuclear non-proliferation regimes) that reliably enforces rules like these, while safe and beneficial AI goes forward. But my view is that even dramatically weaker setups can still help a lo...