The Nonlinear Library LW - Rationality Research Report: Towards 10x OODA Looping? by Raemon
Feb 24, 2024
23:16
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rationality Research Report: Towards 10x OODA Looping?, published by Raemon on February 24, 2024 on LessWrong.
6 months ago I wrote Feedbackloop-first Rationality. I didn't followup on it for awhile (except for sporadic Deliberate ("Purposeful?") Practice Club).
I just spent 6 weeks actually exploring "how would I build my own cognition training program?". In the process of doing so, I've iterated a bunch. I'm still in an orienting phase, but it seemed worth writing down the current stage of my thoughts.
What's my goal?
A rough overview:
I want to get more, higher quality "X-risk thinker hours" hours.
This includes AI alignment technical research, AI macrostrategy research, policy, governance, as well as people (such as Lightcone team) deciding which infrastructure to build,
I'm particularly interested in getting more "serial research", as opposed to more "parallel research." We can throw more researchers at a problem, but if there are some problems that require one person to synthesize 10+ years of experience, all the parallel research won't help.
An obvious way to improve researcher hours is "via mentorship", but I think there is a mentorship bottleneck. So, I'm interested in strategies that train tacit cognitive skills that either don't require mentorship, or leveraging expertise from outside the current x-risk ecosystem.
This is all parented under the higher level goal of "contribute meaningfully to x-risk reduction", but it feels relevant/meaty enough to be worth running at this goal for awhile.
"Rationality for the sake of existential risk"
A part of me romantically wants to pursue "rationality training for rationality training's sake." Alas, the world is big and my time is limited and I just can't actually justify putting years of effort into something, if I didn't think it would help with x-risk.
CFAR went through a phase where (some leaders) framed things as:
"Rationality, for the sake of rationality, for the sake of existential risk."
i.e. try to earnestly build something rationality-focused for it's own sake, because that seemed both healthier and better for x-risk than "rationality for the sake of x-risk", directly.
I think this was a reasonable thing to try, but my impression is this didn't work that well. If you tell yourself (and your students) "I'm doing this for the sake of rationality itself", but then in practice you're getting people to delicately open up their soul and figure out their true goals... and all-the-while radiating "man I really hope your goals turn out to involve saving the worlds from AIs", that may fuck up the "earnestly try to figure out your goals" process.
So:
I am not here to help you earnestly figure out your goals. That's an important part of rationality, and it might come about incidentally while people do exercises I develop, but it's not what I'm focused on this year.
I am here to develop and teach cognitive skills, which help you solve confusing problems at the edge of your ability. I'm doing this to push forward humanity's frontier of "how quickly can we do challenging research?", and strive towards 10x science.
I will prioritize learning and teaching those skills to people who seem like they are going to help with x-risk somehow, but I aim to write up a lot of stuff publicly, and trying-where-possible to output exercises that other people can do on their own, for whatever reasons they want. (See Exercise: Solve "Thinking Physics" as an example)
The Story So Far
Feedback-loops and "deliberate practice", vs "Just Clicking"
I just spent a month workshopping various "teaching rationality" plans. My initial ideas were framed around:
Deliberate practice is costly and kinda sucks
Therefore, people haven't invested in it much, as either "rationality training programs", or as an "alignment research training programs."
Therefore,...
6 months ago I wrote Feedbackloop-first Rationality. I didn't followup on it for awhile (except for sporadic Deliberate ("Purposeful?") Practice Club).
I just spent 6 weeks actually exploring "how would I build my own cognition training program?". In the process of doing so, I've iterated a bunch. I'm still in an orienting phase, but it seemed worth writing down the current stage of my thoughts.
What's my goal?
A rough overview:
I want to get more, higher quality "X-risk thinker hours" hours.
This includes AI alignment technical research, AI macrostrategy research, policy, governance, as well as people (such as Lightcone team) deciding which infrastructure to build,
I'm particularly interested in getting more "serial research", as opposed to more "parallel research." We can throw more researchers at a problem, but if there are some problems that require one person to synthesize 10+ years of experience, all the parallel research won't help.
An obvious way to improve researcher hours is "via mentorship", but I think there is a mentorship bottleneck. So, I'm interested in strategies that train tacit cognitive skills that either don't require mentorship, or leveraging expertise from outside the current x-risk ecosystem.
This is all parented under the higher level goal of "contribute meaningfully to x-risk reduction", but it feels relevant/meaty enough to be worth running at this goal for awhile.
"Rationality for the sake of existential risk"
A part of me romantically wants to pursue "rationality training for rationality training's sake." Alas, the world is big and my time is limited and I just can't actually justify putting years of effort into something, if I didn't think it would help with x-risk.
CFAR went through a phase where (some leaders) framed things as:
"Rationality, for the sake of rationality, for the sake of existential risk."
i.e. try to earnestly build something rationality-focused for it's own sake, because that seemed both healthier and better for x-risk than "rationality for the sake of x-risk", directly.
I think this was a reasonable thing to try, but my impression is this didn't work that well. If you tell yourself (and your students) "I'm doing this for the sake of rationality itself", but then in practice you're getting people to delicately open up their soul and figure out their true goals... and all-the-while radiating "man I really hope your goals turn out to involve saving the worlds from AIs", that may fuck up the "earnestly try to figure out your goals" process.
So:
I am not here to help you earnestly figure out your goals. That's an important part of rationality, and it might come about incidentally while people do exercises I develop, but it's not what I'm focused on this year.
I am here to develop and teach cognitive skills, which help you solve confusing problems at the edge of your ability. I'm doing this to push forward humanity's frontier of "how quickly can we do challenging research?", and strive towards 10x science.
I will prioritize learning and teaching those skills to people who seem like they are going to help with x-risk somehow, but I aim to write up a lot of stuff publicly, and trying-where-possible to output exercises that other people can do on their own, for whatever reasons they want. (See Exercise: Solve "Thinking Physics" as an example)
The Story So Far
Feedback-loops and "deliberate practice", vs "Just Clicking"
I just spent a month workshopping various "teaching rationality" plans. My initial ideas were framed around:
Deliberate practice is costly and kinda sucks
Therefore, people haven't invested in it much, as either "rationality training programs", or as an "alignment research training programs."
Therefore,...