The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The LessWrong 2022 Review: Review Phase, published by RobertM on December 22, 2023 on LessWrong. This year's LessWrong review nomination phase ended a few days ago, with 339 posts nominated. For comparison, 291 posts were nominated in the 2021 review. Nomination Phase Results Here are the current top-20 posts by vote total: AGI Ruin: A List of Lethalities MIRI announces new "Death With Dignity" strategy Simulators Where I agree and disagree with Eliezer Reward is not the optimization target Six Dimensions of Operational Adequacy in AGI Projects You Are Not Measuring What You Think You Are Measuring Epistemic Legibility Let's think about slowing down AI It Looks Like You're Trying To Take Over The World Staring into the abyss as a core life skill Counterarguments to the basic AI x-risk case Sazen Losing the root for the tree The shard theory of human values Limerence Messes Up Your Rationality Real Bad, Yo Models Don't "Get Reward" Toni Kurz and the Insanity of Climbing Mountains Butterfly Ideas On how various plans miss the hard bits of the alignment challenge (I'm sensing a bit of a theme...) More than 60 posts have already been reviewed, but that leaves quite a few posts that have yet to receive any reviews, including many of the most-upvoted ones. If you want to see which posts are most under-reviewed, you can switch your sorting to Magic (Needs Review)[1]. Maybe you have thoughts on Paul's thoughts on Eliezer's thoughts? Inline Reacts! We've got these new nifty inline reacts which you can leave on posts (not just comments!); you may have noticed them. I encourage you to make good use of these when reviewing posts. (Typos should now be a lot less annoying to report, if you're inclined to do so.) Prizes? Prizes! Last year we awarded prizes for good reviews. This year we will also award prizes! We're aiming for something similar to last year's, though we haven't yet worked out the details (size, scope, etc). Final Voting The review phase ends on January 14th, which is when final voting starts. ^ Or click that link! Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety Chatbot, published by markov on December 22, 2023 on LessWrong. Hello World! The AISafety.info team is launching a prototype of the AI Safety Chatbot. The chatbot uses a dataset of alignment literature to answer any questions related to AI safety that you might have, while also citing established sources. Please keep in mind that this is a very early prototype and despite citing references, it may still provide inaccurate or inappropriate information. The overall objective is to help people better understand AI Safety issues based on alignment research using an LLM. This helps with tailoring content to the user's needs and technical level. The chatbot can hopefully be used by both newcomers to AI safety, as well as researchers and engineers who want to get up to speed on specific topics. How it works This chatbot builds upon AlignmentSearch. Our work also expands upon the alignment research dataset (ARD) developed during AI Safety Camp 6. This involved updating and curating the dataset to focus more on quality over quantity. Additionally, we created a process to regularly fetch new articles from selected sources. The ARD contains information about alignment from various books, research papers, and blog posts. For a full list of all the sources being used, look at the readme of the repository on GitHub or HuggingFace. We use a process called retrieval-augmented generation (RAG) to generate the answers. Since LLM data is static, RAG increases the capabilities of a LLM by referencing an external authoritative knowledge base before generating a response. So the process can be roughly broken into - 1) getting and storing the data in a vector database, and then 2) generating an answer based on that data. The information storage process is outlined below: Source: DeepLearning.AI (2023) " LangChain: Chat with Your Data" Document Loading: The articles are scraped from various sources such as the ones mentioned above. They are then parsed and stored in an SQL database while making sure that metadata values fields are valid. Splitting: Then the text content of the documents is broken up into fixed-sized chunks. Storage: These chunks are then embedded into the Pinecone vector database using the OpenAI embedding model. Once we have a database of alignment literature, we use the following series of steps to generate an answer based on a user query: Source: DeepLearning.AI (2023) " LangChain: Chat with Your Data" Query: A user types in a question. Storage+Retrieval: We retrieve chunks from the vector database that are semantically similar to the user's question. Prompt: A prompt is formed that includes all the text retrieved from the relevant chunks provided as context, along with additional instructions on how to format citations and structure the answer. Output: This prompt is then passed to the LLM, which synthesizes an answer based on the relevant chunk of data along with accurate inline citations to the source material. Additionally, as the answer is generated, a ' glossary' is injected with manually written one-sentence definitions of common jargon. The following image example shows what Goodhart's Law looks like on hover: With automatic updates, the ARD will periodically fetch new article entries from trusted sources and add or update items to a SQL database. A separate process adds text to the dataset from user suggested sources. This dataset is available on HuggingFace, which includes instructions on how to download and use it. This means that the chatbot will always be able to produce the more relevant and newer information. We are also experimenting with multiple modes for different audiences. Currently, we offer three options, which produce answers of varying complexity, using the same chunks but adjusting the prompt sent to the LLM. Hallucinations Each chun...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On OpenAI's Preparedness Framework, published by Zvi on December 21, 2023 on LessWrong. Previously: On RSPs. Be Prepared OpenAI introduces their preparedness framework for safety in frontier models. A summary of the biggest takeaways, which I will repeat at the end: I am very happy the preparedness framework exists at all. I am very happy it is beta and open to revision. It's very vague and needs fleshing out in several places. The framework exceeded expectations, with many great features. I updated positively. I am happy we can talk price, while noting our prices are often still far apart. Critical thresholds seem too high, if you get this wrong all could be lost. The High threshold for autonomy also seems too high. The framework relies upon honoring its spirit and not gaming the metrics. There is still a long way to go. But that is to be expected. There is a lot of key detail that goes beyond that, as well. Anthropic and OpenAI have now both offered us detailed documents that reflect real and costly commitments, and that reflect real consideration of important issues. Neither is complete or adequate in its current form, but neither claims to be. I will start with the overview, then go into the details. Both are promising, if treated as foundations to build upon, and if the requirements and alarms are honored in spirit rather than treated as technical boxes to be checked. The study of frontier AI risks has fallen far short of what is possible and where we need to be. To address this gap and systematize our safety thinking, we are adopting the initial version of our Preparedness Framework. It describes OpenAI's processes to track, evaluate, forecast, and protect against catastrophic risks posed by increasingly powerful models. Very good to acknowledge up front that past efforts have been inadequate. I also appreciate this distinction: Three different tasks, in order, with different solutions: Make current models well-behaved. Guard against dangers from new frontier models. Prepare for the endgame of superintelligent AI systems. What works best on an earlier problem likely will not work on a later problem. What works on a later problem will sometimes but not always also solve an earlier problem. I also appreciate that the framework is labeled as a Beta, and that it is named a Preparedness Framework rather than an RSP (Responsible Scaling Policy, the name Anthropic used that many including myself objected to as inaccurate). Basic Principles Their approach is, like many things at OpenAI, driven by iteration. Preparedness should be driven by science and grounded in facts We are investing in the design and execution of rigorous capability evaluations and forecasting to better detect emerging risks. In particular, we want to move the discussions of risks beyond hypothetical scenarios to concrete measurements and data-driven predictions. We also want to look beyond what's happening today to anticipate what's ahead. This is so critical to our mission that we are bringing our top technical talent to this work. We bring a builder's mindset to safety Our company is founded on tightly coupling science and engineering, and the Preparedness Framework brings that same approach to our work on safety. We learn from real-world deployment and use the lessons to mitigate emerging risks. For safety work to keep pace with the innovation ahead, we cannot simply do less, we need to continue learning through iterative deployment. There are big advantages to this approach. The biggest danger in the approach is the potential failure to be able to successfully anticipate what is ahead in exactly the most dangerous situations where something discontinuous happens. Another danger is that if the safety requirements are treated as check boxes rather than honored in spirit, then it is easy to optimi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Most People Don't Realize We Have No Idea How Our AIs Work, published by Thane Ruthenis on December 21, 2023 on LessWrong. This point feels fairly obvious, yet seems worth stating explicitly. Those of us familiar with the field of AI after the deep-learning revolution know perfectly well that we have no idea how our ML models work. Sure, we have an understanding of the dynamics of training loops and SGD's properties, and we know how ML models' architectures work. But we don't know what specific algorithms ML models' forward passes implement. We have some guesses, and some insights painstakingly mined by interpretability advances, but nothing even remotely like a full understanding. And most certainly, we wouldn't automatically know how a fresh model trained on a novel architecture that was just spat out by the training loop works. We're all used to this state of affairs. It's implicitly-assumed shared background knowledge. But it's actually pretty unusual, when you first learn of it. And... I'm pretty sure that the general public doesn't actually know that. They still think in GOFAI terms. They still believe that all of an AI's functionality has been deliberately programmed, not trained, into it. That behind every single thing ChatGPT can do, there's a human who implemented that functionality and understands it. Or, at the very least, that it's written in legible, human-readable and human-understandable format, and that we can interfere on it in order to cause precise, predictable changes. Polls already show concern about AGI. If the fact that we don't know what these systems are actually thinking were widely known and properly appreciated? If there weren't the implicit assurance of "someone understands how it works and why it can't go catastrophically wrong"? Well, I expect much more concern. Which might serve as a pretty good foundation for further pro-AI-regulations messaging. A way to acquire some political currency you can spend. So if you're doing any sort of public appeals, I suggest putting the proliferation of this information on the agenda. You get about five words (per message) to the public, and "Powerful AIs Are Black Boxes" seems like a message worth sending out.[1] ^ There's been some pushback on the "black box" terminology. I maintain that it's correct: ML models are black boxes relative to us, in the sense that by default, we don't have much more insight into what algorithms they execute than we'd have by looking at a homomorphically-encrypted computation to which we don't have the key, or by looking at the activity of a human brain using neuroimaging. There's been a nonzero amount of interpretability research, but it's still largely the case; and would be almost fully the case for models produced by novel architectures. ML models are not black boxes relative to the SGD, yes. The algorithm can "see" all computations happening, and tightly intervene on them. But that seems like a fairly counter-intuitive use of the term, and I maintain that "AIs are black boxes" conveys all the correct intuitions. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prediction Markets aren't Magic, published by SimonM on December 21, 2023 on LessWrong. One common theme which I come across quite a bit in the prediction market space is: Prediction markets would solve [x][1] And the proposal for "solving" [x] is: Set up prediction market ??? Profit These people need to consider the idea that "prediction markets aren't as popular as think think because they aren't as good as you think". (And I say this as a person who is a big fan of prediction markets!) If you think prediction markets are valuable it's likely because you think they price things well - probably due to some kind of market efficiency... well why hasn't that efficiency led to the creation of prediction markets... Where are all the prediction markets? Maybe if prediction markets aren't popular for your specific usecase, it's because prediction markets are less efficient. The cost to markets of acquiring information is high Prediction markets are very good at enabling a diverse group of participants to ensemble their forecasts in sensible ways. However, they are not very good at compensating participants[2]. Simple example - all information from same source For example, consider a market on a coin-flip, with some unknown probability p of heads. The market will resolve based on the outcome of a single coin flip. However, the coin is available for anyone else to come over and test, but there's a catch. You have to pay to flip the coin. How many times would you flip the coin? To make this simplified model even simpler, lets assume that participants will always take as much profit from the market as possible (eg they are risk neutral or the size of the market is small relative to their bank-roll). Under these assumptions, after each flip the partipants will move the market price to their new posterior. Well, after n flips the market price is going to be μn=1nni=11ith flip is success (this will depend on the initial prior, we can do all these calculations explicitly with a beta distribution but it doesn't alter the result). How much should we expect this to move by paying for an additional sample? So we should expect to move the mean by O(1n), therefore our pnl will be will be O(1n2)[3]. So people will keep collecting samples for the market while costn2>liquidity. Therefore we can see that roughly speaking we will obtain O(liquiditycost) samples. But this is strictly much worse than if rather than seeding the market the liquidity provider just went out and collected liquiditycost samples. One other thing to notice about this model of the prediction market is that early participants benefit much more than later participants. (This appears to be a general "problem" with markets where the subsidies accrue to the fastest players, rather than those adding the most information[4]). Additional theoretical justification In our first example, we have given all the advantages to the market. There is one source of information, it is passed immediately between all participants (if there was only one participant the market would work just as well), the cost of collecting data is known upfront. Any duplication of effort is inefficient from the point of view of the market subsidizer. From the point of view of any participant, their participation must be EV positive (in effort terms), but their EV must be equal to the EV lost by the market subsider. Therefore any duplication of effort must be a direct cost born by the subsidiser. Concrete Example - Manifold.Love To come back to the example which convinced me this article needed writing: Manifold.Love. I am assuming you're familiar with the premise. "Dating app powered by prediction markets". My (simplified) model for dating apps, is roughly speaking: Collect data on users (pictures, profile text, age, gender, location etc) Collect more data ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New positions on Open Philanthropy's Cause Prioritization team (Global Health and Wellbeing), published by Open Philanthropy on December 21, 2023 on The Effective Altruism Forum. Open Philanthropy plans to grant more than $300 million per year to causes in our Global Health and Wellbeing (GHW) portfolio over the next few years. We're hiring for two types of roles (Research and Strategy Fellows) on the GHW portfolio's Cause Prioritization team, which works closely with senior leadership and program officers to conduct research that improves our grantmaking and high-level strategy.[1] The team's work includes: Investigating potential new cause areas Evaluating and prioritizing across existing cause areas Advancing research agendas within existing cause areas Contributing to high-level strategy decisions Partnering with other organizations and philanthropists to advance the practice of cost-effective grantmaking To illustrate what these roles involve day-to-day, here are a few recent projects managed by Research and Strategy Fellows on the GHW Cause Prioritization team: In 2021, we announced hires to lead our grantmaking in global aid advocacy and South Asian air quality, two new cause areas we added as a result of the team's research. In 2022, we hired program officers in global health R&D and effective altruism community building (global health and wellbeing), again based on the team's research and early grantmaking. In 2022, we ran the Regranting Challenge, a $150 million initiative to fund highly effective teams at other grantmaking organizations, and the Cause Exploration Prizes (with support from our 2022 summer interns), where we invited people to suggest new areas for us to support. In 2023, based on the team's research, we announced a new program area: Global Public Health Policy, including grantmaking on lead exposure, alcohol policy, and suicide prevention. We conduct shallow- and medium-depth investigations as part of our work to explore new potential cause areas. Two examples of shallow investigations: Telecommunications in LMICs and Civil Conflict Reduction . The team is fully remote; you can work from anywhere (time zones permitting - see the listing for more). And these positions don't require specialized experience - though we are especially interested in candidates who have experience living or working in low- and middle-income countries. To see more detail on the roles, and to apply, visit the job listing. To learn more about working at Open Philanthropy, visit our careers page. And please feel free to share any questions in a comment, or by emailing jobs@openphilanthropy.org. ^ Note that these roles would not have a significant focus on our Farm Animal Welfare grantmaking, though this is included in the GHW portfolio. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: It is called Effective Altruism, not Altruistic Effectiveness, published by Timon Renzelmann on December 21, 2023 on The Effective Altruism Forum. This post is a personal reflection on certain attitudes I have encountered in the EA community that I believe can be misleading. It is primarily based on intuition, not thorough research and surveys. It is not news that the EA community has an unbalanced demographic, with men in the majority. I have heard from several women what they dislike about the EA community and this post is what I have taken from those conversations. I think that if we can move more in the direction I'm describing, the EA community can become warmer and more welcoming to all genders and races (and also more effective at doing good). I'd like to note that I don't think what I'm about to describe is a widespread problem, but a phenomenon that may occur in some places. Most of my experiences with the EA community have been very positive. I meet mostly caring people with whom I can have interesting, sometimes controversial discussions. And I often meet people who are very willing to help. Now to the subject: Some women I have spoken to have described a "lack of empathy" in the group, or, more specifically, that EA people came across as "tech bros" who lacked humility and wouldn't help a stranger because it wouldn't be the most effective thing to do. In an introductory discussion group we ran (in our university group), one of the participants perceived some of EA's ideas as "cold-hearted" and was very critical of the abstract, sometimes detached way of trying to calculate how to do good most effectively. I believe that these impressions and experiences point to risks associated with certain EA-related ideas. The idea of optimization Firstly, the idea of optimising/maximising one's impact is fraught with risks, which have been described already here, here and here (and maybe elsewhere, too). To judge between actions or causes as more or less worthy of our attention can certainly seem cold-hearted. While this approach is valuable for triage and for prioritising in difficult situations, it also has a dark side when it justifies not caring about what we might normally care about. We should not discredit what might be judged as lesser goods just because some metric suggests it. It shouldn't lead us to lose our humility (impacts are uncertain and we are not omniscient) as well as our sense of caring. What kind of community are we if people don't feel comfortable talking about their private lives because they don't optimise everything, don't spend their free time researching or trying to make a difference? When people think that spending time volunteering for less effective non-profits might not be valued or even dismissed? What is the point of an ineffective soup kitchen, after all it is a waste of time in terms of improving QALYs? I have no doubt that even the thought of encountering such insensitive comments makes you feel uncomfortable. The following quote might appear to conflict with the goal of EA, but I think it doesn't and makes and important point. "There is no hierarchy of compassionate action. Based on our interests, skills and what truly moves us, we each find our own way, helping to alleviate suffering in whatever way we can." - Joseph Goldstein (2007) in A Heart Full of Peace What we are trying to do is called Effective Altruism, not Altruistic Effectiveness, and we should be trying to be altruistic in the first place, that is, good and caring people.[1] The idea of focusing on consequences I also think that an exaggerated focus on consequences can be misleading in a social context, as well as detrimental in terms of personal well-being. Even if one supports consequentialism, focusing on consequences may not be the best strategy for achieving the...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Legalize butanol?, published by bhauth on December 21, 2023 on LessWrong. ethanol Alcoholic drinks are popular in most of the world. Excessive consumption of them is also a major public health problem. Bans have been attempted, sometimes successfully, sometimes unsuccessfully, but some people argue that alcohol plays a necessary role in social interactions. Alcoholic drinks contain ethanol, which is metabolized to acetaldehyde, which is metabolized to acetate. In cells, ethanol is mostly unreactive but can bind to receptors. Acetaldehyde reacts with lots of stuff, mostly reversibly but sometimes irreversibly. Small amounts of acetate are essentially irrelevant, mostly providing calories. Acetaldehyde can inactivate enzymes by causing crosslinking. Large amounts of it are generally bad. We can separate out the effects of ethanol itself and acetaldehyde by looking at people who metabolize acetaldehyde slowly. About 50% of people of Northeast Asian descent have a dominant mutation in their acetaldehyde dehydrogenase gene, making this enzyme less effective, which causes the alcohol flush reaction, also known as Asian flush syndrome. A similar mutation is found in about 5-10% of blond-haired blue-eyed people of Northern European descent. In these people, acetaldehyde accumulates after drinking alcohol, leading to symptoms of acetaldehyde poisoning, including the characteristic flushing of the skin and increased heart and respiration rates. Other symptoms can include severe abdominal and urinary tract cramping, hot and cold flashes, profuse sweating, and profound malaise. Individuals with deficient acetaldehyde dehydrogenase activity are far less likely to become alcoholics, but seem to be at a greater risk of liver damage, alcohol-induced asthma, and contracting cancers of the oro-pharynx and esophagus due to acetaldehyde overexposure. Wikipedia alternatives to ethanol Ethanol is what's in drinks because it's produced naturally by a common type of fermentation, it prevents growth of most harmful microbes, and the yeast produced has some nutritional value. But our modern industrial civilization is no longer bound by such prosaic concerns. Can we do better? ether Studies, including that of an ether addict in 2003, have shown that ether causes dependence; however, the only symptom observed was a will to consume more ether. No withdrawal symptoms were prevalent. Wikipedia Diethyl ether has the same direct effect as ethanol, but mostly isn't metabolized in the body. Some of it gets metabolized (by a monooxygenase) by oxidation to (ethanol + acetaldehyde), but more of it gets exhaled. Thus, it's similar to what ethanol without acetaldehyde production would be like. Diethyl ether isn't expensive to make, and there's lots of knowledge about its effects because it was widely consumed in the past. But it does have some problems: It's volatile and has a strong smell, so it's obnoxious to other people. It has fairly low water solubility, ~6%. Above 2% in air, it's inflammable. Pure diethyl ether exposed to oxygen can slowly form explosive peroxides. It's already been banned most places, and unbanning things might be harder than not banning them. butanol At sub-lethal doses, 1-butanol acts as a depressant of the central nervous system, similar to ethanol: one study in rats indicated that the intoxicating potency of 1-butanol is about 6 times higher than that of ethanol, possibly because of its slower transformation by alcohol dehydrogenase. Wikipedia Some butanol occurs naturally in fermented products. Yeasts could be engineered to produce mostly butanol instead of ethanol, but the maximum practical concentration from fermentation is low, ~1%. If it's 6x as effective as ethanol, then 1% would be enough for drinks. It would then provide a similar effect to ethanol with less aldehyde pr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The privilege of native English speakers in reaching high-status, influential positions in EA, published by Alix Pham on December 21, 2023 on The Effective Altruism Forum. Huge thanks to Konrad Seifert, Marcel Steimke, Ysaline Bourgine, Milena Canzler, Alex Rahl-Kaplan, Marieke de Visscher, and Guillaume Vorreux for the valuable feedback provided on drafts of this post, and to many others for the conversations that lead to me writing it. Views & mistakes are my own. TL;DR Being a non-native English speaker makes one sound less convincing. However, poor inclusion of non-native English speakers means missed perspectives in decision-making. Hence, it's a vicious circle where lack of diversity persists: native English culture prevails at the thought leadership level and neglects other cultures by failing to acknowledge that it is inherently harder to stand out as a non-native English speaker. Why I am writing this I'm co-directing EA Switzerland (I'm originally from France), and I've been thinking about the following points for some time. I've been invited to speak at the Panel on Community Building at EAG Boston 2023, where I shared a rougher version of those thoughts. I was pretty scared to share this in a place where the vast majority of attendees matched the description "native English speaker", but after talking to a few people, it felt true. Many of the non-native speakers related, and many of the native speakers acknowledged it. steerers). I'm pretty scared to share it here too of course, but it's probably worth it. An unconscious bias against non-native English speakers The beauty of linguistic diversity is that it reveals to us just how ingenious and how flexible the human mind is. Human minds have invented not one cognitive universe, but 7,000. Lera Boroditsky Non-native English speakers sound less convincing The neural pathways that form in your brain during childhood will affect how you think as an adult. Depending on where and with which languages and cultures you grew up, the conceptual space in which your brain processes and communicates information will be different. Then, when a non-native speaker expresses their thoughts and opinions in English, most times it will be lower-fidelity than native speakers, and will probably be less convincing and/or sound less smart.[1] Besides, I can relate to the experience shared here that native English speakers are sometimes hard to follow when your own native language (and culture) is not English.[2] I guess it's especially true when your English is good enough that it doesn't appear necessary to speak slower - but I think for most of us, it still is necessary to speak slower or repeat stuff, and avoid referencing local pop culture. Usually though, non-native speakers would like to avoid asking to slow down, repeat, or clarify because, on top of being burdensome, it can be associated with incompetence. Hence, it's important not to confuse competence with language proficiency, and keep in mind that for the majority of non-native English speakers, it's harder to engage with the materials, harder to understand and intervene in debates, and harder to speak and write with fidelity to one's thoughts. As a consequence, it's then harder to be understood, stand out, get hired, and get heard. A similar case has been made for less-STEM-than-average people in EA.[3] Additionally, the English language and vocabulary might also not allow one to express the full length of their thoughts - words might not even exist for them. Different languages can allow for different profiles of available concepts and thoughts, because their structure and vocabulary vary. Poor inclusion of non-native English speakers means missed perspectives One could consider the 6 dimensions of culture as a good illustration of the effect of culture (and then, ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On the future of language models, published by Owen Cotton-Barratt on December 21, 2023 on The Effective Altruism Forum. 1. Introduction 1.1 Summary of key claims Even without further breakthroughs in AI, language models will have big impacts in the coming years, as people start sorting out proper applications The early important applications will be automation of expert advisors, management, and perhaps software development The more transformative but harder prizes are automation of research and automation of executive capacity In their most straightforward form ("foundation models"), language models are a technology which naturally scales to something in the vicinity of human-level (because it's about emulating human outputs), not one that naturally shoots way past human-level performance i.e. it is a mistake-in-principle to imagine projecting out the GPT-2 - GPT-3 - GPT-4 capability trend into the far-superhuman range Although they're likely to be augmented by things which accelerate progress, this still increases the likelihood of a relatively slow takeoff - several years (rather than weeks or months) of transformative growth before truly wild things are happening seems plausible NB version of "speed superintelligence" could still be transformative even while performance on individual tasks is still firmly human level There are two main techniques which can be used (probably in conjunction) to get language models to do more powerful things than foundation models are capable of: Scaffolding: structured systems to provide appropriate prompts, including as a function of previous answers Finetuning: altering model weights to select for task performance on a particular task Each of these techniques has a path to potentially scale to strong superintelligence; alternatively language models might at some point be obsoleted by another form of AI Timelines for any of these things seem pretty unclear From a safety perspective, language model agents whose agency comes from scaffolding look greatly superior than ones whose agency comes from finetuning Because you can get an extremely high degree of transparency by construction Finetuning is more likely an important tool for instilling virtues (e.g. honesty) in systems Sutton's Bitter Lesson raises questions for this strategy, but needn't mean it's doomed to be outcompeted On the likely development trajectory there are a number of distinct existential risks e.g. guarding against takeover from early language model agents is pretty different from differential technological development to ensure that we automate safety-enhancing research before risk-increasing research The current portfolio of work on AI risk is over-indexed on work which treats "transformative AI" as a black box and tries to plan around that. I think that we can and should be peering inside that box (and this may involve plans targeted at more specific risks). 1.2 Meta We know that AI is likely to be a very transformative technology. But a lot of the analysis of this point treats something like "AGI" as a black box, without thinking too much about the underlying tech which gets there. I think that's a useful mode, but it's also helpful to look at specific forms of AI technology and ask where they're going and what the implications are. This doc does that for language models. It's a guide for thinking about them from various angles with an eye to what the strategic implications might be. Basically I've tried to write the thing I wish I'd read a couple of years ago; I'm sharing now in case it's helpful for others. The epistemic status of this is "I thought pretty hard about this and these are my takes"; I'm sure there are still holes in my thinking (NB I don't actually do direct work with language models), and I'd appreciate pushback; but I'm also pretty sure I'm ...