The Nonlinear Library

Critically examining ROKO's claims about an engineered pandemic, exploring the origins of COVID-19 in Wuhan, controversies over ROKO's assertions, and analyzing pandemic origins and ROKO's claims with a focus on virology research and live animal markets.

Exploring the issues with Gemini 1.5 AI's image generation, revealing inaccuracies and biases. Discussing the controversy of AI-generated images depicting incorrect subjects. Highlighting the challenges of AI alignment, bias, and the importance of AI safety measures. Delving into the partisan political split and bias issues within Gemini, raising concerns about alignment with specific political leanings. Examining the risks and complexities of AI deception, emphasizing the need for ethical handling of AI models.

A discussion on recent AI advancements including Gemini Pro 1.5 and Sora. OpenAI's ChatGPT glitched causing chaos, Google's Gemini image generator had systematic errors, and the ethical implications of AI systems. The episode also covers AI forecasting, language model complexities, child behavior modification, job outlook in AI, regulations, and risks in AI development.

The podcast discusses how large language models struggle to understand sensory aspects of language, presented through the H-Test benchmark. Key findings include plateauing performance with stronger models and minimal impact of example quantity.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sora What, published by Zvi on February 23, 2024 on LessWrong. Hours after Google announced Gemini 1.5, OpenAI announced their new video generation model Sora. Its outputs look damn impressive. How Sora Works How does it work? There is a technical report. Mostly it seems like OpenAI did standard OpenAI things, meaning they fed in tons of data, used lots of compute, and pressed the scaling button super hard. The innovations they are willing to talk about seem to be things like 'do not crop the videos into a standard size.' That does not mean there are not important other innovations. I presume that there are. They simply are not talking about the other improvements. We should not underestimate the value of throwing in massively more compute and getting a lot of the fiddly details right. That has been the formula for some time now. Some people think that OpenAI was using a game engine to learn movement. Sherjil Ozair points out that this is silly, that movement is learned easily. The less silly speculation is that game engine outputs may have been in the training data. Jim Fan thinks this is likely the case, and calls the result a 'data-driven physics engine.' Raphael Molière thinks this is likely, but more research is needed. Brett Goldstein here digs into what it means that Sora works via 'patches' that combine to form the requested scene. Gary Marcus keeps noting how the model gets physics wrong in various places, and, well, yes, we all know, please cut it out with the Stop Having Fun. Yishan points out that humans also work mostly on 'folk physics.' Most of the time humans are not 'doing logic' they are vibing and using heuristics. I presume our dreams, if mapped to videos, would if anything look far less realistic than Sora. Yann LeCun, who only a few days previous said that video like Sora produces was not something we knew how to do, doubled down with the ship to say that none of this means the models 'understand the physical world,' and of course his approach is better because it does. Why update? Is all of this technically impressive? Sora Is Technically Impressive Yes, Sora is definitely technically impressive. It was not, however, unexpected. Sam Altman: we'd like to show you what sora can do, please reply with captions for videos you'd like to see and we'll start making some! Eliezer Yudkowsky: 6 months left on this timer. Eliezer Yudkowsky (August 26, 2022): In 2-4 years, if we're still alive, anytime you see a video this beautiful, your first thought will be to wonder whether it's real or if the AI's prompt was "beautiful video of 15 different moth species flapping their wings, professional photography, 8k, trending on Twitter". Roko (other thread): I don't really understand why anyone is freaking out over Sora. This is entirely to be expected given the existence of generative image models plus incrementally more hardware and engineering effort. It's also obviously not dangerous (in a "take over the world" sense). Eliezer Yudkowsky: This is of course my own take (what with having explicitly predicted this). But I do think you want to hold out a space for others to say, "Well *I* didn't predict it, and now I've updated." Altman's account spent much of last Thursday making videos for people's requests, although not so many that they couldn't cherry pick the good ones. As usual, there are failures that look stupid, mistakes 'a person would never make' and all that. And there are flashes of absolute brilliance. How impressive? There are disputes. Tom Warren: this could be the "holy shit" moment of AI. OpenAI has just announced Sora, its text-to-video AI model. This video isn't real, it's based on a prompt of "a cat waking up its sleeping owner demanding breakfast…" Daniel Eth: This isn't impressive. The owner doesn't wake up, so the AI clearly didn't understa...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contra Ngo et al. "Every 'Every Bay Area House Party' Bay Area House Party", published by Ricki Heicklen on February 23, 2024 on LessWrong. With thanks to Scott Alexander for the inspiration, Jeffrey Ladish, Philip Parker, Avital Morris, and Drake Thomas for masterful cohosting, and Richard Ngo for his investigative journalism. Last summer, I threw an Every Bay Area House Party themed party. I don't live in the Bay, but I was there for a construction-work-slash-webforum-moderation-and-UI-design-slash-grantmaking gig, so I took the opportunity to impose myself on the ever generous Jeffrey Ladish and host a party in his home. Fortunately, the inside of his house is already optimized to look like a parody of a Bay Area house party house, so not much extra decorating was needed, but when has that ever stopped me? Richard Ngo recently covered the event, with only very minor embellishments. I've heard rumors that some people are doubting whether the party described truly happened, so I'd like to set the record straight. Thus, this is part linkpost, part exegesis, and part shameless promotion of my events for potential future venue-lenders. The party had 60 attendees, at least according to the Manifold Market on the topic. Upon arrival, partygoers put on tags with their name, professional title, and LessWrong karma. Attendees were also instructed to put on a wristband that successfully communicated their flirting policies. I took the wristband for people who glomarize about their flirting preferences; Richard took the wristband for those who flirt with all and only those who don't flirt with themselves. Richard writes: You scan around the living room, trying to figure out who to talk to first. The host is sitting on the sofa, with two boxes attached to the front of her shirt. One is filled with money, the other empty. A guy next to her is surreptitiously one-boxing, but she presciently slaps his hand away without even looking. This is defamation. The second box wasn't (necessarily) empty, and Richard certainly never got the opportunity to look inside it. You might be wondering what was in the box. Unfortunately for you, I glomarize not only about my flirting policies but also about my box contents. He is correct, though, that I managed to fend off all the surreptitious one-boxers, with the exception of my girlfriend Avital. She still doesn't know the contents - I would never date someone irresponsible enough to let unknown entities out of a box. The party was PYOL (provide your own liquidity), but we did offer two punches: one for "Contextualizers," and one for "Decouplers or Homophobes". Avital and I drank the punch for "Decouplers or Homophobes." We're still coupled, so you can come to your own conclusions about how homophobic we must be. My favorite part of the night happened when a circle formed around a Jewish Orthodox Rabbi friend of mine who had never heard of Rationality or Effective Altruism. Everyone at the party was eager to give him context. I joined the circle as they were discussing expanding moral circles, and the relative weight of animal lives and future people. "Eliezer doesn't even think cows are sentient," one attendee was saying. "But shrimp are!" another interrupted, causing the group to crack up. "What?" my Rabbi friend said. "Okay, back up, how much Peter Singer have you read?" another attendee said. "Like, have you read Animal Liberation and The Expanding Circle, or just Famine, Affluence, and Morality?" Avital and I tried to explain to them that our friend had not heard of any of the people they were naming, but they didn't seem to understand. Avital turned to me and said "I guess when you've read The Sequences too many times you forget about inferential distance." Eventually, my Rabbi friend said "Okay, so what I'm hearing is: you're expected to t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Do sparse autoencoders find "true features"?, published by Demian Till on February 22, 2024 on LessWrong. In this post I'll discuss an apparent limitation of sparse autoencoders (SAEs) in their current formulation as they are applied to discovering the latent features within AI models such as transformer-based LLMs. In brief, I'll cover the following: I'll argue that the L1 regularisation used to promote sparsity when training SAEs may cause neurons in the sparse layer to learn to represent common combinations of features rather than the individual features that we want them to discover As well as making it more difficult to understand what the actual latent features are, I'll also argue that this limitation may result in some less common latent features not being discovered at all, not even within combinations I'll then explain why I think that the phenomenon of feature splitting observed in Anthropic's SAE paper appears to demonstrate that this limitation does indeed have a large impact on the features discovered by SAEs Finally I'll propose an approach for overcoming this limitation and discuss how we can test whether it really brings us closer to finding the real latent features Rough definition of "true features" We intend for SAEs to discover the "true features" (a term I'm borrowing from Anthropic's SAE paper) used by the target model e.g. a transformer-based LLM. There isn't a universally accepted definition of what "true features" are, but for now I'll use the term somewhat loosely to refer to something like: linear directions in an activation space at a hidden layer within a target model which encode some reasonably monosemantic quantity such as the model's "confidence" in some concept being in play they should play a causal role in the functioning of the target model. So for example if we were to activate or deactivate the feature while the target model is processing a given input sequence then we should expect the outputs to change accordingly in some reasonably understandable way they should be in their most atomic form, so that e.g an arbitrary linear combination of two "true feature" directions is not necessarily itself a "true feature" direction even though it may satisfy the previous criteria There may be other ways of thinking about features but this should give us enough to work with for our current purposes. Why SAEs are incentivised to discover combinations of features rather than individual features Consider a toy setup where one of the hidden layers in the target model has 3 "true features" represented by the following directions in its activation space: Additionally, suppose that feature 1 and feature 2 occur far more frequently than feature 3, and that all features can potentially co-occur in a given activation vector. For the sake of simplicity let's also suppose for now that when features 1 & 2 occur together they tend to both activate with some roughly fixed proportions. For example, an activation vector in which both features 1 and 2 are present (but not feature 3) might look like the following: Now suppose we train an SAE with 3 neurons in the sparse layer on activation vectors from this hidden layer such as the one above. The desirable outcome is that each of the 3 neurons in the sparse layer learns one of the 3 "true features". If this happens then the directions learnt by SAE would mirror the directions of the "true features" in the target model, looking something like: However depending on the respective frequencies of feature 3 vs features 1 & 2, as well as the value of the L1 regularisation weight, I will argue shortly that what may happen is that two of the neurons learn to detect when each of features 1 & 2 respectively occur by themselves, while the third neuron learns to detect when they both occur together. In this case the di...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Job Listing: Managing Editor / Writer, published by Gretta Duleba on February 22, 2024 on LessWrong. MIRI is hiring a managing editor and one or more writers. Here's the job listing. We're accepting applications until March 13th. I am the Communications Manager at MIRI and I'm the hiring manager for these positions. I'm happy to answer questions. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: From salt intake reduction to labor migration: Announcing top ideas for the AIM 2024 CE Incubation Program, published by CE on February 22, 2024 on The Effective Altruism Forum. TLDR: In this post, we announce our top four charity ideas to launch through our August 12-October 4, 2024 Incubation Program. They are the result of months of work by our research team, who selected them through a five-stage research process. We pick interventions that exceed ambitious cost-effectiveness bars, have a high quality of evidence, minimal failure modes, and high expected value. We're also announcing cause areas we'll investigate for the February-March 2025 IP. We're seeking people to launch these ideas through our next Incubation Program. No particular previous experience is necessary - if you could plausibly see yourself excited to launch one of these charities, we encourage you to apply. The deadline for applications is April 14, 2024. You can apply to both August-October 2024 or February-March 2025 programs via the same link below: In the Incubation Program, we provide two months of cost-covered training, stipends (£1900/month during and for up to two months after the program), seed funding up to $200,000, operational support in the initial months, co-working space at our CE office in London, ongoing mentorship, and access to a community of alumni, funders, and experts. Learn more on our CE Incubation Program page. One sentence summaries Advocacy for salt intake reduction An organization seeking to convince governments and the food industry to lower the content of salt in food by setting sodium limits and reformulating high-sodium foods, thereby improving cardiovascular health. Facilitating international labor migration via a digital platform An organization that would facilitate the international migration of workers from low- and middle-income countries using a transparent digital platform paired with low-cost personalized support. Ceramic filters for improving water quality An organization focused on reducing the incidence of diarrhea and other waterborne illnesses by providing free ceramic water filters to families without access to clean drinking water. Participatory Learning and Action (PLA) groups for maternal and newborn health An organization focused on improving newborn and maternal health in rural villages by training facilitators and running PLA groups - a specific type of facilitated self-help group. One-paragraph Summaries Advocacy for salt intake reduction High salt consumption contributes to poor cardiovascular health. Worldwide, cardiovascular diseases are the leading cause of death and are among the top ten contributors to years lived with disabilities. There is good evidence that reducing the amount of sodium people consume in their diets can reduce the risk of cardiovascular problems. Therefore, several countries have successfully reduced the sodium intake of their population, for example by setting sodium limits, which led to the food industry reformulating certain high-salt products. We think that an organization advocating and assisting in implementing these policies could cost-effectively improve the health of millions. Facilitating international labor migration via a digital platform Migrating abroad for work can bring huge financial benefits to people experiencing poverty. Millions of people every year try to tap into these benefits by applying for temporary jobs in higher-income countries. However, the market for matching candidates with jobs is often highly inefficient and riddled with misinformation, putting candidates at financial and personal risk. In many countries, it can cost candidates several years' worth of salaries to secure a job abroad. Fraud is also highly prevalent, leading to candidates often failing to migrate and instead ending up i...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The One and a Half Gemini, published by Zvi on February 22, 2024 on LessWrong. Previously: I hit send on The Third Gemini, and within half an hour DeepMind announced Gemini 1.5. So this covers Gemini 1.5. One million tokens, and we are promised overall Gemini Advanced or GPT-4 levels of performance on Gemini Pro levels of compute. This post does not cover the issues with Gemini's image generation, and what it is and is not willing to generate. I am on top of that situation and will get to it soon. One Million Tokens Our teams continue pushing the frontiers of our latest models with safety at the core. They are making rapid progress. In fact, we're ready to introduce the next generation: Gemini 1.5. It shows dramatic improvements across a number of dimensions and 1.5 Pro achieves comparable quality to 1.0 Ultra, while using less compute. It is truly bizarre to launch Gemini Advanced as a paid service, and then about a week later announce the new Gemini Pro 1.5 is now about as good as Gemini Advanced. Yes, actually, I do feel the acceleration, hot damn. And that's not all! This new generation also delivers a breakthrough in long-context understanding. We've been able to significantly increase the amount of information our models can process - running up to 1 million tokens consistently, achieving the longest context window of any large-scale foundation model yet. One million is a lot of tokens. That covers every individual document I have ever asked an LLM to examine. That is enough to cover my entire set of AI columns for the entire year, in case I ever need to look something up, presumably Google's NotebookLM is The Way to do that. A potential future 10 million would be even more. Soon Gemini will be able to watch a one hour video or read 700k words, whereas right now if I use the web interface of Gemini Advanced interface all I can upload is a photo. The standard will be to give people 128k tokens to start, then you can pay for more than that. A million tokens is not cheap inference, even for Google. Oriol Vinyals (VP of R&D DeepMind): Gemini 1.5 has arrived. Pro 1.5 with 1M tokens available as an experimental feature via AI Studio and Vertex AI in private preview. Then there's this: In our research, we tested Gemini 1.5 on up to 2M tokens for audio, 2.8M tokens for video, and 10M tokens for text. From Shannon's 1950s bi-gram models (2 tokens), and after being mesmerized by LSTMs many years ago able to model 200 tokens, it feels almost impossible that I would be talking about hundreds of thousands of tokens in context length, let alone millions. Jeff Dean (Chief Scientist, Google DeepMind): Multineedle in haystack test: We also created a generalized version of the needle in a haystack test, where the model must retrieve 100 different needles hidden in the context window. For this, we see that Gemini 1.5 Pro's performance is above that of GPT-4 Turbo at small context lengths and remains relatively steady across the entire 1M context window, while the GPT-4 Turbo model drops off more quickly (and cannot go past 128k tokens). Guido Appenzeller (responding to similar post): Is this really done with a monolithic model? For a 10M token window, input state would be many Gigabytes. Seems crazy expensive to run on today's hardware. Sholto Douglas (DeepMind): It would honestly have been difficult to do at decent latency without TPUs (and their interconnect) They're an under appreciated but critical piece of this story Here are their head-to-head results with themselves: Here is the technical report. There is no need to read it, all of this is straightforward. Their safety section says 'we followed our procedure' and offers no additional details on methodology. On safety performance, their tests did not seem to offer much insight, scores were similar to Gemini Pro 1.0. Mixture...