The Nonlinear Library LW - Why I think it's net harmful to do technical safety research at AGI labs by Remmelt
Feb 7, 2024
02:23
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why I think it's net harmful to do technical safety research at AGI labs, published by Remmelt on February 7, 2024 on LessWrong.
IMO it is harmful on expectation for a technical safety researcher to work at DeepMind, OpenAI or Anthropic.
Four reasons:
Interactive complexity. The intractability of catching up - by trying to invent general methods for AI corporations to somehow safely contain model interactions, as other engineers scale models' combinatorial complexity and outside connectivity.
Safety-capability entanglements
Commercialisation. Model inspection and alignment techniques can support engineering and productisation of more generally useful automated systems.
Infohazards. Researching capability risks within an AI lab can inspire researchers hearing about your findings to build new capabilities.
Shifts under competitive pressure
DeepMind merged with Google Brain to do commercialisable research, OpenAI set up a company and partnered with Microsoft to release ChatGPT,
Anthropic pitched to investors they'd build a model 10 times more capable.
If you are an employee at one of these corporations, higher-ups can instruct you to do R&D you never signed up to do.[1] You can abide, or get fired.
Working long hours surrounded by others paid like you are, by a for-profit corp, is bad for maintaining bearings and your epistemics on safety.[2]
Safety-washing. Looking serious about 'safety' helps labs to recruit idealistic capability researchers, lobby politicians, and market to consumers.
'let's build AI to superalign AI'
'look, pretty visualisations of what's going on inside AI'
This is my view. I would want people to engage with the different arguments, and think for themselves what ensures that future AI systems are actually safe.
^
I heard via via that Google managers are forcing DeepMind safety researchers to shift some of their hours to developing Gemini for product-ready launch.
I cannot confirm whether that's correct.
^
For example, I was in contact with a safety researcher at an AGI lab who kindly offered to read my comprehensive outline on the AGI control problem, to consider whether to share with colleagues. They also said they're low energy. They suggested I'd remind them later, and I did, but they never got back to me. They're simply too busy it seems.
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
IMO it is harmful on expectation for a technical safety researcher to work at DeepMind, OpenAI or Anthropic.
Four reasons:
Interactive complexity. The intractability of catching up - by trying to invent general methods for AI corporations to somehow safely contain model interactions, as other engineers scale models' combinatorial complexity and outside connectivity.
Safety-capability entanglements
Commercialisation. Model inspection and alignment techniques can support engineering and productisation of more generally useful automated systems.
Infohazards. Researching capability risks within an AI lab can inspire researchers hearing about your findings to build new capabilities.
Shifts under competitive pressure
DeepMind merged with Google Brain to do commercialisable research, OpenAI set up a company and partnered with Microsoft to release ChatGPT,
Anthropic pitched to investors they'd build a model 10 times more capable.
If you are an employee at one of these corporations, higher-ups can instruct you to do R&D you never signed up to do.[1] You can abide, or get fired.
Working long hours surrounded by others paid like you are, by a for-profit corp, is bad for maintaining bearings and your epistemics on safety.[2]
Safety-washing. Looking serious about 'safety' helps labs to recruit idealistic capability researchers, lobby politicians, and market to consumers.
'let's build AI to superalign AI'
'look, pretty visualisations of what's going on inside AI'
This is my view. I would want people to engage with the different arguments, and think for themselves what ensures that future AI systems are actually safe.
^
I heard via via that Google managers are forcing DeepMind safety researchers to shift some of their hours to developing Gemini for product-ready launch.
I cannot confirm whether that's correct.
^
For example, I was in contact with a safety researcher at an AGI lab who kindly offered to read my comprehensive outline on the AGI control problem, to consider whether to share with colleagues. They also said they're low energy. They suggested I'd remind them later, and I did, but they never got back to me. They're simply too busy it seems.
Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org