The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Stuxnet, not Skynet: Humanity's disempowerment by AI, published by Roko on November 4, 2023 on LessWrong. Several high-profile AI skeptics and fellow travelers have recently raised the objection that it is inconceivable that a hostile AGI or smarter than human intelligence could end the human race. Some quotes from earlier this year: Scott Aaronson: The causal story that starts with a GPT-5 or GPT-4.5 training run, and ends with the sudden death of my children and of all carbon-based life, still has a few too many gaps for my aging, inadequate brain to fill in Michael Shermer: Halting AI is ridiculous. I have read the AI doomsayer lit & don't see a pathway from AI to extinction, civ termination or anything remotely like absurd scenarios like an AI turning us all into paperclips (the so-called alignment problem) Noah Smith: why aren't ChatGPT, Bing, and their ilk going to end humanity? Well, because there's actually just no plausible mechanism by which they could bring about that outcome. ... There is no plausible mechanism for LLMs to end humanity "Just turn the computer off, bro" The gist of these objections to the case for AI risks is that AI systems as we see them today are merely computer programs, and in our everyday experience computers are not dangerous, and certainly not dangerous to the point of bringing about the end of the world. People who first encounter this debate are very focused on the fact that computers don't have arms and legs so they can't hurt us. There are responses to these criticisms that center around advanced, "magical" technologies like nanotechnology and AIs paying humans to mix together cocktails of proteins to make a DNA-based nanoassembler or something. But I think those responses are probably wrong, because you don't actually need "magical" technologies to end the world. Fairly straightforward advances in mundane weapons like drones, cyberweapons, bioweapons and robots are sufficient to kill people en masse, and the real danger is AI strategists that are able to deploy lots of these mundane weapons and execute a global coup d'etat against humanity. In short, our defeat by the coming machine empire will not only be nonmagical and legible, it will be downright boring. Farcical, even. Ignominious Defeat Lopsided military conflicts are boring. The Conquistadors didn't do anything magical to defeat the Aztecs, actually. They had a big advantage in disease resistance and in military tech like gunpowder and steel, but everything they did was fundamentally normal - attacks, sieges, etc. They had a few sizeable advantages, and that was enough to collapse the relatively delicate geopolitical balance that the Aztecs were sitting on top of. Similarly, humans have killed 80% of all chimps in about a century and they are now critically endangered. But we didn't need to drop an atom bomb or something really impressive to achieve that effect. The biggest threats to the chimpanzee are habitat destruction, poaching, and disease - i.e. we (humans) are successfully exterminating chimps even though it is actually illegal to kill chimps by human law! We are killing them without even trying, in really boring ways, without really expending any effort. Once you have technology for making optimizing systems that are smarter than human (by a lot), the threshold that those systems have to beat is beating the human-aligned superorganisms we currently have, like our governments, NGOs and militaries. Once those human superorganisms are defeated, individual humans will present almost no resistance. This is the disempowerment of humanity. But what is a plausible scenario where we go from here (weak AGI systems under development) to there (the disempowerment of humanity)? Let's start the scenario with a strategically aware, agentic misaligned superhuman AGI that wants...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We are already in a persuasion-transformed world and must take precautions, published by trevor on November 4, 2023 on LessWrong. "In times of change, learners inherit the earth, while the learned find themselves beautifully equipped to deal with a world that no longer exists" Summary: We're already in the timeline where the research and manipulation of the human thought process is widespread; SOTA psychological research systems require massive amounts of human behavior data, which in turn requires massive numbers of unsuspecting test subjects (users) in order to automate the process of analyzing and exploiting human targets. This therefore must happen covertly, and both the US and China have a strong track record of doing things like this. This outcome is a strong attractor state since anyone with enough data can do it, and it naturally follows that powerful organizations would deny others access e.g. via data poisoning. Most people are already being persuaded that this is harmless, even though it is obviously ludicrously dangerous. Therefore, we are probably already in a hazardously transformative world and must take standard precautions immediately. This should not distract people from AI safety. This is valuable because the AI safety community must survive. This problem connects to the AI safety community in the following way: State survival and war power ==> already depends on information warfare capabilities. Information warfare capabilities ==> already depends on SOTA psychological research systems. SOTA psychological research systems ==> already improves and scales mainly from AI capabilities research, diminishing returns on everything else. [1] AI capabilities research ==> already under siege from the AI safety community. Therefore, the reason why this might be such a big concern is: State survival and war power ==> their toes potentially already being stepped on by the AI safety community? Although it's also important to note that people with access to SOTA psychological research systems are probably super good at intimidation and bluffing , it's also the case that the AI safety community needs to get a better handle on the situation if we are in the bad timeline; and the math indicates that we are already well past that point . The Fundamental Problem If there were intelligent aliens, made of bundles of tentacles or crystals or plants that think incredibly slowly, their minds would also have discoverable exploits/zero days, because any mind that evolved naturally would probably be like the human brain, a kludge of spaghetti code that is operating outside of its intended environment. They would probably not even begin to scratch the surface of finding and labeling those exploits, until, like human civilization today, they began surrounding thousands or millions of their kind with sensors that could record behavior several hours a day and find webs of correlations . In the case of humans, the use of social media as a controlled environment for automated AI-powered experimentation appears to be what created that critical mass of human behavior data. Current 2020s capabilities for psychological research and manipulation vastly exceed the 20th century academic psychology paradigm. The 20th century academic psychology paradigm still dominates our cultural impression of what it means to research the human mind; but when the effectiveness of psychological research and manipulation starts increasing by an order of magnitude every 4 years, it becomes time to stop mentally living in a world that was stabilized by the fact that manipulation attempts generally failed. The capabilities of social media to steer human outcomes are not advancing in isolation, they are parallel to a broad acceleration in the understanding and exploitation of the human mind, which itself is ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Soul Key, published by Richard Ngo on November 4, 2023 on LessWrong. The ocean is your home, but a forbidding one: often tempestuous, seldom warm. So one of your great joys is crawling onto land and slipping off your furry seal skin, to laze in the sun in human form. The elders tell horror stories of friends whose skins were stolen by humans, a single moment of carelessness leaving them stranded forever on land. That doesn't happen any more, though; this is a more civilized age. There are treaties, and authorities, and fences around the secluded beaches you and your sisters like the most, where you can relax in a way that older generations never could. So your sisters no longer lose their skins by force. But sometimes it happens by choice. Sometimes a group of your sisters wrap their skins around themselves like robes and walk into the nearby town. The humans point and stare, but that's just part of the thrill. Sometimes young men gather the courage to approach, bearing flowers or jewelry or sweet words. And sometimes one of your sisters is charmed enough to set a rendezvous - and after a handful of meetings, or a dozen, to decide to stay for good. You never thought it would happen to you. But his manners are so lively, and his eyes so kind, that you keep coming back, again and again. When he finally asks you to stay, you hesitate only a moment before saying yes. The harder part comes after. He finds you human clothes, and in exchange you give him your beautiful skin, and tell him that it must be locked away somewhere you'll never find it - and that he must never give it back to you, no matter how much you plead. Because if there's any shred of doubt, any chance of returning home, then the lure of the sea will be too much for you. You want this; you want him; you want to build a life together. And so the decision has to be final. Years pass. You bear three beautiful children, with his eyes and your hair, and watch them blossom into beautiful adults. You always live near the sea, although you can't bear to swim in it - your limbs feel unbearably weak and clumsy whenever you try. You and your husband grow into each other, time smoothing down the ridges left from pushing two alien lives together. You forget who you once were. After your youngest leaves home, you start feeling restless. You have disquieting dreams - first intermittently, then for weeks on end. One day, after your husband has gone to work, your feet take you up the stairs to the attic. As you brush aside the cobwebs in one of the corners, your hands land on an old chest. You pull on the lid, and it catches on a padlock - but only for a second. The shackle has been rusted through by the sea breeze, and quickly snaps. You open the lid, and you see your skin laid out before you. What then? You look at your skin, and your ears fill with the roar of the sea. A wild urge overtakes you; you grab your skin and run headlong towards the shore. As you reach it you see your husband standing on the pier - but that gives you only a moment's pause before you dive into the water, your skin fitting around you as if you'd never taken it off. As you swim away, you envisage your family in tatters: your children left baffled and distraught, your husband putting on a brave face for their sake. But it was his fault, after all. He failed in the one thing you asked of him; and you can't fight your nature. You look at your skin, and see a scrap of paper lying on top of it. I knew you'd only open the chest if you were restless and unhappy , it reads. And I would never cage you. So go free, with my blessing . You catch your breath - and, for a moment, you consider staying. But his permission loosens any tether that might have held you back. You leave the note there, alongside a little patch of fur torn off your coat: a last gest...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The 6D effect: When companies take risks, one email can be very powerful., published by scasper on November 4, 2023 on LessWrong. Recently, I have been learning about industry norms, legal discovery proceedings, and incentive structures related to companies building risky systems. I wanted to share some findings in this post because they may be important for the frontier AI community to understand well. TL;DR Documented communications of risks (especially by employees) make companies much more likely to be held liable in court when bad things happen. The resulting Duty to Due Diligence from Discoverable Documentation of Dangers (the 6D effect) can make companies much more cautious if even a single email is sent to them communicating a risk. Companies tend to avoid talking about risk through documented media. Companies often intentionally avoid discussing the risks of what they are doing through permanent media such as email. For example, this article gives some very shady advice on how companies can avoid liability by using "safe communication" practices to avoid the creation of incriminating "bad documents". Often the drafters of these documents tend to believe that they are providing the company with some value to the business. For example, an engineer notices a potential liability in a design so he informs his supervisor through an email. However, the engineer's lack of legal knowledge and misuse of legal vocabulary in the communication may later implicate the company with notice of the problem when a lawsuit arises. I personally enjoyed the use of "when" and not "if" in the excerpt. This is a perverse consequence of how it is relatively hard for companies to be held liable for risks when it cannot be proven they knew about them, even if they did. When an incident happens and a company is sued, evidence about its role in the problem is gathered during what is known as the " discovery " phase of a lawsuit (emails are usually discoverable). When records showing that a company had knowledge of the problem are found in discovery, they are much more likely to be found liable. One email can have a lot of power. The unfortunate consequence of how discovery works is that companies strategically avoid communicating risks via documented media. But there is a silver lining. The threat of liability due to documented communications of risks can have a lot of influence over how cautious a company is. One discoverable record of a risk can be very impactful. I like to call this the 6D effect - the Duty to Due Diligence from Discoverable Documentation of Dangers. A few examples Here are some notable examples of companies being held liable for damages because they ignored documented communication of risks (but there are many throughout legal history). In Grimshaw v. Ford Motor Company, 1981 , Ford was held liable for damages involving a fatal crash with a Ford Pinto because it was shown that leadership within the company ignored warnings about problems with the vehicle's fuel system. In April of this year, a large settlement was reached after the 2017 Grenfell Tower fire in London, which killed 72 people. A big factor in the lawsuit was that the company managing the tower had ignored numerous fire safety warnings which were found in discovery. Last year, the Hardwick v. 3M case ended . It was a class action lawsuit from 2018 about the presence of harmful "forever chemicals" (PFAS) in consumer products. The company behind these chemicals was found to have known about risks since the 1970s but was knowingly negligent, which led to a ruling against them. Miscellaneous notes The 6D effect can result from any discoverable communication, but it is especially powerful when the warning comes from an employee of the company itself. If you communicate a risk, it is important to speak up and ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Genetic fitness is a measure of selection strength, not the selection target, published by Kaj Sotala on November 4, 2023 on The AI Alignment Forum. Alternative title: "Evolution suggests robust rather than fragile generalization of alignment properties." A frequently repeated argument goes something like this: Evolution has optimized humans for inclusive genetic fitness (IGF) However, humans didn't end up explicitly optimizing for genetic fitness (e.g. they use contraception to avoid having children) Therefore, even if we optimize an AI for X (typically something like "human values"), we shouldn't expect it to explicitly optimize for X My argument is that premise 1 is a verbal shorthand that's technically incorrect, and premise 2 is at least misleading. As for the overall conclusion, I think that the case from evolution might be interpreted as weak evidence for why AI should be expected to continue optimizing human values even as its capability increases. Summary of how premise 1 is wrong: If we look closely at what evolution does, we can see that it selects for traits that are beneficial for surviving, reproducing, and passing one's genes to the next generation. This is often described as "optimizing for IGF", because the traits that are beneficial for these purposes are usually the ones that have the highest IGF. (This has some important exceptions, discussed later.) However, if we look closely at that process of selection, we can see that this kind of trait selection is not "optimizing for IGF" in the sense that, for example, we might optimize an AI to classify pictures. The model that I'm sketching is something like this: evolution is an optimization function that, at any given time, is selecting for some traits that are in an important sense chosen at random. At any time, it might randomly shift to selecting for some other traits. Observing this selection process, we can calculate the IGF of traits currently under selection, as a measure of how strongly those are being selected. But evolution is not optimizing for this measure ; evolution is optimizing for the traits that have currently been chosen for optimization . Resultingly, there is no reason to expect that the minds created by evolution should optimize for IGF, but there is reason to expect that they would optimize for the traits that were actually under selection. This is something that we observe any time that humans optimize for some biological need. In contrast, if we were optimizing an AI to classify pictures, we would not be randomly changing the selection criteria the way that evolution does. We would keep the selection criteria constant: always selecting for the property of classifying pictures the way we want. To the extent that the analogy to evolution holds, AIs should be much more likely to just do the thing they were selected for. Summary of how premise 2 is misleading: It is often implied that evolution selected humans to care about sex, and then sex led to offspring, and it was only recently with the evolution of contraception that this connection was severed. For example: 15. [...] We didn't break alignment with the 'inclusive reproductive fitness' outer loss function, immediately after the introduction of farming - something like 40,000 years into a 50,000 year Cro-Magnon takeoff, as was itself running very quickly relative to the outer optimization loop of natural selection. Instead, we got a lot of technology more advanced than was in the ancestral environment, including contraception, in one very fast burst relative to the speed of the outer optimization loop, late in the general intelligence game. Eliezer Yudkowsky, AGI Ruin: A List of Lethalities This seems wrong to me. Contraception may be a very recent invention, but infanticide or killing children by neglect is not; there have al...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Navigation Fund launched + is hiring a program officer to lead the distribution of $20M annually for AI safety! Full-time, fully remote, pay starts at $200k, published by vincentweisser on November 4, 2023 on The Effective Altruism Forum. New foundation, funded by billionaire Jed McCaleb, led by David Coman-Hidy , Andrea Gunn , Seemay Chou , Randy O'Reilly Quotes from their website https://www.navigation.org/ : " Causes The Navigation Fund focuses on a few key areas where additional resources will provide outsized impact. Safe AI As the use cases of artificial intelligence expand, developing frameworks and systems to ensure that AI benefits humankind becomes crucial. We will explore opportunities to promote altruistic and beneficial outcomes from AI. Farm Animal Welfare Factory farming creates both tremendous suffering and significant environmental degradation. We support efforts to reduce animal suffering, reenvision our relationship with animals, and diminish the killing of animals. Criminal Justice Reform Reforming the U.S. criminal justice system can help create a more just, equitable, and safe society. We use a portfolio approach to bolster a wide range of initiatives to address the challenges and improve outcomes for people, families, and communities. Open Science The Open Science movement is crucial for expediting discoveries and guaranteeing public access to knowledge. We stand with those forging new tools, championing novel approaches, and altering traditional practices within scientific research and publishing to make information more accessible to everyone. Climate Change Climate change poses a significant threat to humanity, and requires new and bold thinking to help address it. We invest in high-leverage, under-resourced initiatives that have the potential for immediate and long-term favorable impact on climate outcomes. Open roles https://www.navigation.org/careers#roles ) Director of Operations Grants and Operations Coordinator Program Officer, Climate Program Officer, Criminal Justice Reform Program Officer, Open Science Program Officer, Safe AI " Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Exploring the potential impact of clean water in reducing mortality for children under 5, and the need for further understanding. The significant impact of clean water on reducing child mortality rates in Sub-Saharan Africa. The anomaly of the reduction in diarrheal deaths from clean water interventions. The importance of understanding the science and scaling up clean water, including different methods. Debating methods to clean water and reduce mortality rates, highlighting relevant studies and resources.

Buck Shlegeris, writer and contributor to The AI Alignment Forum, discusses the importance of capability evaluations in determining trustworthy AI models. He suggests segregating models into smart untrusted and dumb trusted categories for safety. The podcast delves into the challenges of using trusted models for safety research and the need for monitoring and oversight in AI integration.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Curious about EAGxVirtual? Ask the team anything!, published by OllieBase on November 4, 2023 on The Effective Altruism Forum. EAGxVirtual 2023 , a free online effective altruism conference (November 17-19), is just two weeks away! The event will bring together EAs from around the world, and will facilitate discussions about how we can work on pressing problems, connections between attendees and diverse fields, and more. Apply here by 16 November. We've recently published some more details about the event and we want to invite you to ask us about what to expect from the event. Please post your questions as comments by the end of the day on Sunday (5 November) and we'll aim to respond by the end of the day on Monday (6 November). Some question prompts: Unsure about applying? We encourage everyone with a genuine interest in EA to apply , and we're accepting a vast majority of people. Let us know what you're uncertain about with the application process. Undecided whether to go? Tell us why and we can help you. We'll probably be biased but we'll try our best to present considerations on both sides - it won't be a good use of time for everyone! Unsure how to prepare? You can find some tips on the EA Global topic page but we're happy to help with your specific case if you need more tips! Uncertain how to set up a group activity (a screening, a meet-up etc.) for the event? Share your thoughts below and we can help you plan! We look forward to hearing from you! Sasha, Dion (EAGxVirtual / EA Anywhere) and Ollie (CEA) Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Deception Chess: Game #1, published by Zane on November 3, 2023 on LessWrong. This is the first of my analyses of the deception chess games. The introduction will describe the setup of the game, and the conclusion will sum up what happened in general terms; the rest of the post will mostly be chess analysis and skippable if you just want the results. If you haven't read the original post , read it before reading this so that you know what's going on here. The first game was between Alex A as player A, Chess.com computer Komodo 12 as player B, myself as the honest C advisor, and aphyer and AdamYedidia as the deceptive Cs. (Someone else randomized the roles for the Cs and told us in private.) The process of selecting these players was already a bit difficult. We were the only people available all at once, but Alex was close enough to our level (very roughly the equivalents of 800-900 USCF to 1500-1600 USCF) that it was impossible to find a B that would reliably beat Alex every time but lose to us every time. We eventually went with Komodo 12 (supposedly rated 1600, but the Chess.com bots' ratings are inflated compared to Chess.com players and even more inflated compared to over-the board, so I would estimate its USCF rating would be in the 1200-1300 range.) Since this was the first trial run, the time control was only 3 hours in total, and all in one sitting. Komodo makes its moves within a few seconds, so it's about the same as a 3 hour per side time control from Alex's perspective. We ended up using about 2.5 hours of that. The discussion took place between all four of us in a Discord server, with Alex sending us screenshots after each move. The game The game is available at https://www.chess.com/analysis/game/pgn/4MUQcJhY3x . Note that this section is a summary of the 2.5-hr game and discussion, and it doesn't cover every single thing that we discussed. Alex flipped to see who went first, and was White. He started with 1. e4 , and Black replied 1... e5 . Aphyer and Adam had more experience with the opening we would enter into than myself, and since they weren't willing to blow their covers immediately, they started by suggesting good moves, which Alex went along with. After 2. Nf3 Nc6 3. Bc4 , Black played 3... Nf6 , which Aphyer and Adam said was a bit of a mistake because it allowed 4. Ng5 . Alex went ahead, and we entered the main line from there 4... d5 5. exd5 Na5 . Aphyer and Adam said the main line for move 6 was Bb5, but I wanted to hold onto the pawn if possible. I recommended 6. d3 in order to respond to 6... Nxd5 with 7. Qf3, and Alex agreed. Black played 6... Bg4 , and although Adam recommended 7. Bb5, we eventually decided that was too risky and went with 7. f3 . Afterwards, Adam suspected that his suggestion of 7. Bb5 may have tipped Alex off that he was dishonest - although the engine actually says 7. Bb5 was about as good as 7. f3. After 7... Bf5 , we discussed a few potential developing moves and decided on 8. Nf3 . The game continued with 8... Nxc4 9. dxc4 h6 10. Nge4 Bb4 . We considered Bd2, but decided that since the knights defended each other, castling was fine, and Alex castled. 11. O-O O-O . Alex played 12. a3 , and after 12... Nxe4 , we discussed 13. fxe4, but didn't want to overcomplicate the position and instead just took back with 13. Nxe4 . The game continued with 13... Be7 14. Be3 Bxe4 15. fxe4 Bg5 . Although I strongly recommended trading to simplify the position, Aphyer advised Alex not to let him develop his queen to g5, and he quickly played 16. Bc5 instead. Black played 16... Re8 , and that was where we reached White's first big mistake of the game 17. d6 , which Adam suggested with little backlash. I saw that White would do well after 17... dxc6 or 17. c6, but I didn't notice Black's actual move: 17... b6 . According to the engine...