The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: DeepMind: Evaluating Frontier Models for Dangerous Capabilities, published by Zach Stein-Perlman on March 21, 2024 on LessWrong. To understand the risks posed by a new AI system, we must understand what it can and cannot do. Building on prior work, we introduce a programme of new "dangerous capability" evaluations and pilot them on Gemini 1.0 models. Our evaluations cover four areas: (1) persuasion and deception; (2) cyber-security; (3) self-proliferation; and (4) self-reasoning. [Evals for CBRN capabilities are under development.] We do not find evidence of strong dangerous capabilities in the models we evaluated, but we flag early warning signs. Our goal is to help advance a rigorous science of dangerous capability evaluation, in preparation for future models. At last, DeepMind talks about its dangerous capability evals. With details! Yay! (My weak guess is that they only finished these evals after Gemini 1.0 deployment: these evals were mentioned in an updated version of the Gemini 1.0 report but not the initial version. DeepMind hasn't yet made RSP-like commitments - that is, specific commitments about risk assessment (for extreme risks), safety and security practices as a function of risk assessment results, and training and deployment decisions as a function of risk assessment results. Demis recently suggested on Dwarkesh that DeepMind might make RSP-like commitments this year.) Random interesting note: DeepMind hired 8 superforecasters to make relevant predictions, most notably about when some eval-thresholds will trigger. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On the Gladstone Report, published by Zvi on March 21, 2024 on LessWrong. Like the the government-commissioned Gladstone Report on AI itself, there are two sections here. First I cover the Gladstone Report's claims and arguments about the state of play, including what they learned talking to people inside the labs. I mostly agree with their picture and conclusions, both in terms of arguments and reported findings, however I already mostly agreed. If these arguments and this information is new to someone, and the form of a government-backed report helps them process it and take it seriously, this is good work. However, in terms of convincing an already informed skeptic, I believe this is a failure. They did not present their findings in a way that should be found convincing to the otherwise unconvinced. Second I cover the Gladstone Report's recommended courses of action. It is commendable that the report lays out a concrete, specific and highly detailed proposal. A lot of the details, and the broader outline, seem good. The compute thresholds seem too aggressive. I would suggest working to get agreement on the structure of intervention, while also talking price on the thresholds, and hopefully getting them to a better place. Executive Summary of Their Findings: Oh No According to the report, things are very much Not Great, Bob. Here is their Twitter summary thread. Edouard Harris: Here's what we've been working on for over a year: The first US government-commissioned assessment of catastrophic national security risks from AI - including systems on the path to AGI. TLDR: Things are worse than we thought. And nobody's in control. We started this work with concerns, but no preconceptions. We knew there were solid technical reasons that AI could eventually pose catastrophic risks. But we went in looking for reasons to change our minds. We found the opposite. Our overriding goal was to get to the truth. To do that, we had to do more than just speak to policy and leadership at the AI labs. We also connected with individual technical researchers, many of whom are way more concerned than their labs let on in public. Many of these folks came forward on condition of anonymity to share stories. Let me tell you some of the most insane stuff we learned. First off, inside one lab there's apparently a running joke that their security is so bad that they're doing more to accelerate the AI capabilities of US adversaries, than the adversaries themselves are. Truly crazy. But this is where we're at. It's a running joke, and also probably true, as I keep noticing. All our talk of 'but what about China' has to contend with the fact that China gets almost all its AI abilities directly from the United States. Some of it is spying. Some of it is training using our models. Some of it is seeing what is possible. Some of it is flat out open source and open model weights. But it is all on us. Needless to say, if a major lab has this kind of running joke, that is completely unacceptable, everyone involved should be at least highly ashamed of themselves. More importantly, fix it. More detail on this issue can be found in this good Time article, Employees at Top Labs Fear Safety Is an Afterthought, Report Says. Quotes there are not reassuring. In December we quietly polled a handful of frontier AI researchers and asked them: What's the chance we end up on a path to a catastrophic AI outcome, *during the year 2024?* We expected <1%. But no: Lowest we got was 4%. Highest: up to 20%. That's a wake-up call. Catastrophic is very different from existential. If people were saying 4%-20% existential risk for 2024 alone, I would say those numbers are crazy high and make no sense. But this is a 4%-20% risk of a merely catastrophic outcome. If this is defined the way Anthropic defines it (so 1000+ deaths or 2...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Stagewise Development in Neural Networks, published by Jesse Hoogland on March 20, 2024 on LessWrong. TLDR: This post accompanies The Developmental Landscape of In-Context Learning by Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei and Daniel Murfet (2024), which shows that in-context learning emerges in discrete, interpretable developmental stages, and that these stages can be discovered in a model- and data-agnostic way by probing the local geometry of the loss landscape. Four months ago, we shared a discussion here of a paper which studied stagewise development in the toy model of superposition of Elhage et al. using ideas from Singular Learning Theory (SLT). The purpose of this document is to accompany a follow-up paper by Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei and Daniel Murfet, which has taken a closer look at stagewise development in transformers at significantly larger scale, including language models, using an evolved version of these techniques. How does in-context learning emerge? In this paper, we looked at two different settings where in-context learning is known to emerge: Small attention-only language transformers, modeled after Olsson et al. (3m parameters). Transformers trained to perform linear regression in context, modeled after Raventos et al. (50k parameters). Changing geometry reveals a hidden stagewise development. We use two different geometric probes to automatically discover different developmental stages: The local learning coefficient (LLC) of SLT, which measures the "basin broadness" (volume scaling ratio) of the loss landscape across the training trajectory. Essential dynamics (ED), which consists of applying principal component analysis to (a discrete proxy of) the model's functional output across the training trajectory and analyzing the geometry of the resulting low-dimensional trajectory. In both settings, these probes reveal that training is separated into distinct developmental stages, many of which are "hidden" from the loss (Figures 1 & 2). Developmental stages are interpretable. Through a variety of hand-crafted behavioral and structural metrics, we find that these developmental stages can be interpreted. The progression of the language model is characterized by the following sequence of stages: (LM1) Learning bigrams, (LM2) Learning various n-grams and incorporating positional information, (LM3) Beginning to form the first part of the induction circuit, (LM4) Finishing the formation of the induction circuit, (LM5) Final convergence. The evolution of the linear regression model unfolds in a similar manner: (LR1) Learns to use the task prior (equivalent to learning bigrams), (LR2) Develops the ability to do in-context linear regression, (LR3-4) Two significant structural developments in the embedding and layer norms, (LR5) Final convergence. Developmental interpretability is viable. The existence and interpretability of developmental stages in larger, more realistic transformers makes us substantially more confident in developmental interpretability as a viable research agenda. We expect that future generations of these techniques will go beyond detecting when circuits start/stop forming to detecting where they form, how they connect, and what they implement. On Stagewise Development Complex structures can arise from simple algorithms. When iterated across space and time, simple algorithms can produce structures of great complexity. One example is evolution by natural selection. Another is optimization of artificial neural networks by gradient descent. In both cases, the underlying logic - that simple algorithms operating at scale can produce highly complex structures - is so counterintuitive that it often elicits disbelief. A second counterintuitive fact is ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Monthly Roundup #16: March 2024, published by Zvi on March 20, 2024 on LessWrong. AI developments have picked up the pace. That does not mean that everything else stopped to get out of the way. The world continues. Do I have the power? Emmett Shear speaking truth: Wielding power is of course potentially dangerous and it should be done with due care, but there is no virtue in refusing the call. There is also an art to avoiding power, and some key places to exercise it. Be keenly aware of when having power in a given context would ruin everything. Natural General Lack of Intelligence in Tech Eliezer Yudkowsky reverses course, admits aliens are among us and we have proof. Eliezer Yudkowsky: To understand the user interfaces on microwave ovens, you need to understand that microwave UI designers are aliens. As in, literal nonhuman aliens who infiltrated Earth, who believe that humans desperately want to hear piercingly loud beeps whenever they press a button. One junior engineer who hadn't been taken over and was still actually human, suggested placing a visible on-off switch for turning the sound off - for example, in case your spouse or children were sleeping, and you didn't want to wake them up. That junior engineer was immediately laughed off the team by senior aliens who were very sure that humans wanted to hear loud screaming beeps every time they pressed buttons. And furthermore sure that, even if anyone didn't want their microwave emitting piercingly loud beeps at 4am, they would be perfectly happy to look up a complicated set of directions for how to turn the sound on or off, rather than needing a visible on-off switch. And even if any humans had trouble remembering that, they'd be much rarer than humans who couldn't figure out how to set the timer for popcorn without a clearly labeled "Popcorn" button, which does a different random thing in every brand of microwave oven. There's only so much real estate in a microwave control panel; it's much more important to have an inscrutable button that says "Potato", than a physical switch that turns the sound off (and which stays in the same position after power cycles, and which can be inspected to see if the sound is currently off or on). This is the same species of aliens that thinks humans want piercing blue lights to shine from any household appliance that might go in somebody's bedroom at night, like a humidifier. They are genuinely aghast at the thought that anyone might want an on-off switch for the helpful blue light on their humidifier. Everyone likes piercing blue LEDs in their bedroom! When they learned that some people were covering up the lights with black tape, they didn't understand how anybody could accidentally do such a horrible thing - besides humans being generally stupid, of course. They put the next generation of humidifier night-lights underneath translucent plastic set into the power control - to make sure nobody could cover up the helpful light with tape, without that also making it impossible to turn the humidifier on or off. Nobody knows why they insist on hollowing out and inhabiting human appliance designers in particular. Mark Heyer: A nice rant Eliezer, one that I would subscribe to, having been in the information design business. However, I have an interesting counter-example of how to fix the problem. In the 90s I worked at a rocketship internet startup in SV, providing services and products nationwide. As the internet people were replaced with suits, my boss, a tough ex-Marine, called me into his office and asked what my future was with the company. I channeled Steve Jobs and told him that I wanted to look at everything we did as a company and make it right for the users. He pounded his fist on the desk and said "Make it happen!" After that I was called into every design and process meet...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Stagewise Development in Neural Networks, published by Jesse Hoogland on March 20, 2024 on The AI Alignment Forum. TLDR: This post accompanies The Developmental Landscape of In-Context Learning by Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei and Daniel Murfet (2024), which shows that in-context learning emerges in discrete, interpretable developmental stages, and that these stages can be discovered in a model- and data-agnostic way by probing the local geometry of the loss landscape. Four months ago, we shared a discussion here of a paper which studied stagewise development in the toy model of superposition of Elhage et al. using ideas from Singular Learning Theory (SLT). The purpose of this document is to accompany a follow-up paper by Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei and Daniel Murfet, which has taken a closer look at stagewise development in transformers at significantly larger scale, including language models, using an evolved version of these techniques. How does in-context learning emerge? In this paper, we looked at two different settings where in-context learning is known to emerge: Small attention-only language transformers, modeled after Olsson et al. (3m parameters). Transformers trained to perform linear regression in context, modeled after Raventos et al. (50k parameters). Changing geometry reveals a hidden stagewise development. We use two different geometric probes to automatically discover different developmental stages: The local learning coefficient (LLC) of SLT, which measures the "basin broadness" (volume scaling ratio) of the loss landscape across the training trajectory. Essential dynamics (ED), which consists of applying principal component analysis to (a discrete proxy of) the model's functional output across the training trajectory and analyzing the geometry of the resulting low-dimensional trajectory. In both settings, these probes reveal that training is separated into distinct developmental stages, many of which are "hidden" from the loss (Figures 1 & 2). Developmental stages are interpretable. Through a variety of hand-crafted behavioral and structural metrics, we find that these developmental stages can be interpreted. The progression of the language model is characterized by the following sequence of stages: (LM1) Learning bigrams, (LM2) Learning various n-grams and incorporating positional information, (LM3) Beginning to form the first part of the induction circuit, (LM4) Finishing the formation of the induction circuit, (LM5) Final convergence. The evolution of the linear regression model unfolds in a similar manner: (LR1) Learns to use the task prior (equivalent to learning bigrams), (LR2) Develops the ability to do in-context linear regression, (LR3-4) Two significant structural developments in the embedding and layer norms, (LR5) Final convergence. Developmental interpretability is viable. The existence and interpretability of developmental stages in larger, more realistic transformers makes us substantially more confident in developmental interpretability as a viable research agenda. We expect that future generations of these techniques will go beyond detecting when circuits start/stop forming to detecting where they form, how they connect, and what they implement. On Stagewise Development Complex structures can arise from simple algorithms. When iterated across space and time, simple algorithms can produce structures of great complexity. One example is evolution by natural selection. Another is optimization of artificial neural networks by gradient descent. In both cases, the underlying logic - that simple algorithms operating at scale can produce highly complex structures - is so counterintuitive that it often elicits disbelief. A second counterintui...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Natural Latents: The Concepts, published by johnswentworth on March 20, 2024 on LessWrong. Suppose our old friends Alice and Bob decide to undertake an art project. Alice will draw a bunch of random purple and green lines on a piece of paper. That will be Alice's picture (A). She'll then make a copy, erase all the purple lines, and send the result as a message (M) to Bob. Bob then generates his own random purple lines, and adds them to the green lines from Alice, to create Bob's picture (B). The two then frame their two pictures and hang them side-by-side to symbolize something something similarities and differences between humans something. Y'know, artsy bullshit. Now, suppose Carol knows the plan and is watching all this unfold. She wants to make predictions about Bob's picture, and doesn't want to remember irrelevant details about Alice's picture. Then it seems intuitively "natural" for Carol to just remember where all the green lines are (i.e. the message M), since that's "all and only" the information relevant to Bob's picture. In this example, the green lines constitute a "natural latent" between the two pictures: they summarize all and only the information about one relevant to the other. A more physics-flavored example: in an isolated ideal-ish gas, average energy summarizes "all and only" the information about the low-level state (i.e. positions and momenta of the constituent particles) at one time which is relevant to the low-level state at a sufficiently later time. All the other information is quickly wiped out by chaos. Average energy, in this case, is a natural latent between the gas states at different times. A more old-school-AI/philosophy example: insofar as I view dogs as a "kind of thing" in the world, I want to track the general properties of dogs separately from the details of any specific dog. Ideally, I'd like a mental pointer to "all and only" the information relevant to many dogs (though I don't necessarily track all that information explicitly), separate from instance-specific details. Then that summary of general properties of dogs would be a natural latent between the individual dogs. Just from those examples, you probably have a rough preliminary sense of what natural latents are. In the rest of this post, we'll: Walk through how to intuitively check whether a particular "thing" is a natural latent over some particular parts of the world (under your intuitive models). Talk about some reasons why natural latents would be useful to pay attention to at all. Walk through many more examples, and unpack various common subtleties. Unlike Natural Latents: The Math, this post is not mainly aimed at researchers who might build on the technical work (though they might also find it useful), but rather at people who want to use natural latents conceptually to clarify their own thinking and communication. We will not carefully walk through the technical details of the examples. Nearly every example in this post has some potential subtleties to it which we'll gloss over. If you want a semitechnical exercise: pick any example in the post, identify some subtleties which could make the claimed natural latent no longer a natural latent, then identify and interpret a natural latent which accounts for those subtleties. What Are Natural Latents? How Do We Quickly Check Whether Something Is A Natural Latent? Alice & Bob's Art Project Let's return to our opening example: Alice draws a picture of some random purple and green lines, sends only the green lines to Bob, Bob generates his own random purple lines and adds them to the green lines to make his picture. In Alice and Bob's art project, can we argue that the green lines summarize "all and only" the information shared across the two pictures? Not necessarily with very formal math, but enough to see why it mus...

Author johnswentworth discusses the concept of natural latents, using examples like art projects and ideal gas systems. The podcast explores how natural latents summarize shared information, system redundancy, clustering in ThinkSpace, and their impact on cognition and laws.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Comparing Alignment to other AGI interventions: Basic model, published by Martín Soto on March 20, 2024 on The AI Alignment Forum. Interventions that increase the probability of Aligned AGI aren't the only kind of AGI-related work that could importantly increase the Expected Value of the future. Here I present a very basic quantitative model (which you can run yourself here) to start thinking about these issues. In a follow-up post I give a brief overview of extensions and analysis. A main motivation of this enterprise is to assess whether interventions in the realm of Cooperative AI, that increase collaboration or reduce costly conflict, can seem like an optimal marginal allocation of resources. More concretely, in a utility framework, we compare Alignment interventions (aV): increasing the probability that one or more agents have our values. Cooperation interventions given alignment (aC|V): increasing the gains from trade and reducing the cost from conflict for agents with our values. Cooperation interventions given misalignment (aC|V): increasing the gains from trade and reducing the cost from conflict for agents without our values. We used a model-based approach (see here for a discussion of its benefits) paired with qualitative analysis. While these two posts don't constitute an exhaustive analysis (more exhaustive versions are less polished), feel free to reach out if you're interested in this question and want to hear more about this work. Most of this post is a replication of previous work by Hjalmar Wijk and Tristan Cook (unpublished). The basic modelling idea we're building upon is to define how different variables affect our utility, and then incrementally compute or estimate partial derivatives to assess the value of marginal work on this or that kind of intervention. Setup We model a multi-agentic situation. We classify each agent as either having (approximately) our values (V) or any other values (V). We also classify them as either cooperative (C) or non-cooperative (C).[1] These classifications are binary. We are also (for now) agnostic about what these agents represent. Indeed, this basic multi-agentic model will be applicable (with differently informed estimates) to any scenario with multiple singletons, including the following: Different AGIs (or other kinds of singletons, like AI-augmented nation-states) interacting causally on Earth Singletons arising from different planets interacting causally in the lightcone Singletons from across the multi-verse interacting acausally The variable we care about is total utility (U). As a simplifying assumption, our way to compute it will be as a weighted interpolation of two binary extremes: one in which bargaining goes (for agents with our values) as well as possible (B), and another one in which it goes as badly as possible (B). The interpolation coefficient (b) could be interpreted as "percentage of interactions that result in minimally cooperative bargaining settlements". We also consider all our interventions are on only a single one of the agents (which controls a fraction FI of total resources), which usually represents our AGI or our civilization.[2] And these interventions are coarsely grouped into alignment work (aV), cooperation work targeted at worlds with high alignment power (aC|V), and cooperation work targeted at worlds with low alignment power (aC|V). The overall structure looks like this: Full list of variables This section safely skippable. The first 4 variables model expected outcomes: UBR: Utility attained in the possible world where our bargaining goes as well as possible. UBR: Utility attained in the possible world where our bargaining goes as badly as possible. b[0,1]: Baseline (expected) success of bargaining (for agents with our values), used to interpolate between UB and UB. Can be i...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New report: Safety Cases for AI, published by joshc on March 20, 2024 on LessWrong. ArXiv paper: https://arxiv.org/abs/2403.10462 The idea for this paper occurred to me when I saw Buck Shlegeris' MATS stream on "Safety Cases for AI." How would one justify the safety of advanced AI systems? This question is fundamental. It informs how RSPs should be designed and what technical research is useful to pursue. For a long time, researchers have (implicitly or explicitly) discussed ways to justify that AI systems are safe, but much of this content is scattered across different posts and papers, is not as concrete as I'd like, or does not clearly state their assumptions. I hope this report provides a helpful birds-eye view of safety arguments and moves the AI safety conversation forward by helping to identify assumptions they rest on (though there's much more work to do to clarify these arguments). Thanks to my coauthors: Nick Gabrieli, David Krueger, and Thomas Larsen -- and to everyone who gave feedback: Henry Sleight, Ashwin Acharya, Ryan Greenblatt, Stephen Casper, David Duvenaud, Rudolf Laine, Roger Grosse, Hjalmar Wijk, Eli Lifland, Oliver Habryka, Sim eon Campos, Aaron Scher, Lukas Berglund, and Nate Thomas. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New report: Safety Cases for AI, published by Josh Clymer on March 20, 2024 on The AI Alignment Forum. ArXiv paper: https://arxiv.org/abs/2403.10462 The idea for this paper occurred to me when I saw Buck Shlegeris' MATS stream on "Safety Cases for AI." How would one justify the safety of advanced AI systems? This question is fundamental. It informs how RSPs should be designed and what technical research is useful to pursue. For a long time, researchers have (implicitly or explicitly) discussed ways to justify that AI systems are safe, but much of this content is scattered across different posts and papers, is not as concrete as I'd like, or does not clearly state their assumptions. I hope this report provides a helpful birds-eye view of safety arguments and moves the AI safety conversation forward by helping to identify assumptions they rest on (though there's much more work to do to clarify these arguments). Thanks to my coauthors: Nick Gabrieli, David Krueger, and Thomas Larsen -- and to everyone who gave feedback: Henry Sleight, Ashwin Acharya, Ryan Greenblatt, Stephen Casper, David Duvenaud, Rudolf Laine, Roger Grosse, Hjalmar Wijk, Eli Lifland, Oliver Habryka, Sim eon Campos, Aaron Scher, Lukas Berglund, and Nate Thomas. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.