The Nonlinear Library LW - Integrity in AI Governance and Advocacy by habryka
Nov 3, 2023
32:40
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Integrity in AI Governance and Advocacy, published by habryka on November 3, 2023 on LessWrong.
Ok, so we both had some feelings about the recent
Conjecture post on "lots of people in AI Alignment are lying"
, and the
associated marketing campaign and stuff
.
I would appreciate some context in which I can think through that, and also to share info we have in the space that might help us figure out what's going on.
I expect this will pretty quickly cause us to end up on some broader questions about how to do advocacy, how much the current social network around AI Alignment should coordinate as a group, how to balance advocacy with research, etc.
Feelings about Conjecture post:
Lots of good points about people not stating their full beliefs messing with the epistemic environment and making it costlier for others to be honest.
The lying and cowardice frames feel off to me.
I personally used to have a very similar rant to Conjecture. Since moving to DC, I'm more sympathetic to governance people. We could try to tease out why.
The post exemplifies a longterm gripe I have with Conjecture's approach to discourse & advocacy, which I've found pretty lacking in cooperativeness and openness
(Note: I worked there for ~half a year.)
Questions on my mind:
How open should people motivated by existential risk be? (My shoulder model of several people says "take a portfolio approach!" - OK, then what allocation?)
How advocacy-y should people be? I want researchers to not have to tweet their beliefs 24/7 so they can actually get work done
How do you think about this, Oli?
How sympathetic to be about governance people not being open about key motivations and affiliations
I personally used to have a very similar rant to Conjecture. I'm now more sympathetic to governance people. We could try to tease out why.
This direction seems most interesting to me!
My current feelings in the space are that I am quite sympathetic to some comms-concerns that people in government have and quite unsympathetic to some other stuff, and I would also like to clarify for myself where the lines here are.
Curious whether you have any key set of observations or experiences you had that made you more sympathetic.
Observations
I've heard secondhand of at least one instance where a person brought up x risk, then their Congressional office took them less seriously. Other staffers have told me talking about x risk wouldn't play well (without citing specific evidence, but I take their opinions seriously).
(This didn't update me a ton though. My model already included "most people will think this is weird and take you less seriously". The question is, "Do you make it likelier for people to do good things later, all things considered by improving their beliefs, shifting the Overton window, or convincing 1/10 people, etc.?")
I've also personally found it tricky to talk about takeover & existential risks, just because these ideas take a long time to explain, and there are many inferential steps between there and the policies I'm recommending. So, I'm often tempted to mention my x risk motivations only briefly, then focus on whatever's inferentially closest and still true. (Classically, this would be "misuse risks, especially from foreign adversaries and terrorists" and "bioweapon and cyberoffensive capabilities coming in the next few years".)
Separate point which we might want to discuss later
A thing I'm confused about is:
Should I talk about inferentially close things that makes them likeliest to embrace the policies I'm putting on their desk,
Or
, should I just bite the bullet of being confusing and start many meetings with "I'm deeply concerned about humanity going extinct in the next decade because of advancing AI which might try to take over the world. It's a lot to explain but the scientists are on my side. Please ...
Ok, so we both had some feelings about the recent
Conjecture post on "lots of people in AI Alignment are lying"
, and the
associated marketing campaign and stuff
.
I would appreciate some context in which I can think through that, and also to share info we have in the space that might help us figure out what's going on.
I expect this will pretty quickly cause us to end up on some broader questions about how to do advocacy, how much the current social network around AI Alignment should coordinate as a group, how to balance advocacy with research, etc.
Feelings about Conjecture post:
Lots of good points about people not stating their full beliefs messing with the epistemic environment and making it costlier for others to be honest.
The lying and cowardice frames feel off to me.
I personally used to have a very similar rant to Conjecture. Since moving to DC, I'm more sympathetic to governance people. We could try to tease out why.
The post exemplifies a longterm gripe I have with Conjecture's approach to discourse & advocacy, which I've found pretty lacking in cooperativeness and openness
(Note: I worked there for ~half a year.)
Questions on my mind:
How open should people motivated by existential risk be? (My shoulder model of several people says "take a portfolio approach!" - OK, then what allocation?)
How advocacy-y should people be? I want researchers to not have to tweet their beliefs 24/7 so they can actually get work done
How do you think about this, Oli?
How sympathetic to be about governance people not being open about key motivations and affiliations
I personally used to have a very similar rant to Conjecture. I'm now more sympathetic to governance people. We could try to tease out why.
This direction seems most interesting to me!
My current feelings in the space are that I am quite sympathetic to some comms-concerns that people in government have and quite unsympathetic to some other stuff, and I would also like to clarify for myself where the lines here are.
Curious whether you have any key set of observations or experiences you had that made you more sympathetic.
Observations
I've heard secondhand of at least one instance where a person brought up x risk, then their Congressional office took them less seriously. Other staffers have told me talking about x risk wouldn't play well (without citing specific evidence, but I take their opinions seriously).
(This didn't update me a ton though. My model already included "most people will think this is weird and take you less seriously". The question is, "Do you make it likelier for people to do good things later, all things considered by improving their beliefs, shifting the Overton window, or convincing 1/10 people, etc.?")
I've also personally found it tricky to talk about takeover & existential risks, just because these ideas take a long time to explain, and there are many inferential steps between there and the policies I'm recommending. So, I'm often tempted to mention my x risk motivations only briefly, then focus on whatever's inferentially closest and still true. (Classically, this would be "misuse risks, especially from foreign adversaries and terrorists" and "bioweapon and cyberoffensive capabilities coming in the next few years".)
Separate point which we might want to discuss later
A thing I'm confused about is:
Should I talk about inferentially close things that makes them likeliest to embrace the policies I'm putting on their desk,
Or
, should I just bite the bullet of being confusing and start many meetings with "I'm deeply concerned about humanity going extinct in the next decade because of advancing AI which might try to take over the world. It's a lot to explain but the scientists are on my side. Please ...