The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What are the biggest conceivable wins for animal welfare by 2025?, published by Nathan Young on December 11, 2023 on The Effective Altruism Forum. In a years time, let's imagine we live in the 95th percentile world for animal welfare. What wins are there? I am trying to write an article about what the end of 2024 will look like, but I don't know enough about animal welfare. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: You can have an hour of my time, to work on your biggest problem, for free., published by John Salter on December 11, 2023 on The Effective Altruism Forum. Who the fuck are you? I run EA's biggest volunteer organisation. We train psychology graduate volunteers to treat mental illnesses, especially in LMICs. To lead by example, I don't take a salary despite working >50Hs per week. To pay the bills, I coach rich people's children to be happier and more productive. While it funds my living expenses, it's not very impactful. I'm hoping to start serving EAs to fix that. EA stuff I've done Authored or co-authored ~$350 000 of successful grant applications for EA charities Grew my org from 1 person to ~60 FTEs in the first 3 months post-launch Now treating one case of depression / anxiety / phobia for ~$50 on the margin (although, just ~ 1000 clients a year right now; planning to treat 13 000 for ~$20 on the margin by 2025) Trained coaches who've helped ~100 EAs overcome social anxiety, depression, procrastination and other barriers to being happily productive. I played on hard difficulty No relevant connections Cause area for which EAs give few shits Bottom 10% of familial income between age 13 and 21 Shoestring budget to start charity Not extraordinarily smart or hardworking Lost three of the prior five years, before starting the charity, to depression I raise this because it's likely that disproportionate amount of my success is due to my decision-making, as opposed to my circumstances or character, and is thus replicable. People I think could be a good fit Early career EAs, especially entrepreneurs and people with leadership ambitions University students struggling to get the most out of their time People who know they are being held back by psychological issues (e.g. fear / risk aversion / procrastination / anxiety / depression / lack of discipline / bad habits) Anyone interested in entering mental health as a cause area How the hour would work Tell me what you'd like to make progress on and we work on it directly via Zoom. Based on the value provided, decide if you want to continue as a paying client. If so, pay by the session (no contracts etc). If not, no hard feels. ~80% of people who chat with me for an hour decide to hire me on a session by session basis thereafter, sticking around for ~9 months on average. How much would you charge afterwards? Full-time EA coaches charge ~$300 per hour I'm going to start out at $80 per hour. I'd only raise it for new clients thereafter. Relevant Links Website for my charity: https://www.overcome.org.uk/ LinkedIn: https://www.linkedin.com/in/john-salter-b685181ba/ To book the free hour https://calendar.app.google/N1iBRnPHEBis8NXy5 If no time works, but you're really keen to give it a go, dm me and I'll see what I can do. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quick thoughts on the implications of multi-agent views of mind on AI takeover, published by Kaj Sotala on December 11, 2023 on The AI Alignment Forum. There was conversation on Facebook over an argument that any sufficiently complex system, whether a human, a society, or an AGI, will be unable to pursue a unified goal due to internal conflict among its parts, and that this should make us less worried about "one paperclipper"-style AI FOOM scenarios. Here's a somewhat edited and expanded version of my response: 1) yes this is a very real issue 2) yet as others pointed out, humans and organizations are still able to largely act as if they had unified goals, even if they often also act contrary to those goals 3) there's a lot of variance in how unified any given human is. trauma makes you less unified, while practices such as therapy and certain flavors of meditation can make a person significantly more unified than they used to be. if you were intentionally designing a mind, you could create mechanisms that artificially mimicked the results of these practices 4) a lot of the human inconsistency looks like it has actually been evolutionarily adaptive for social purposes. E.g. if your social environment punishes you for having a particular trait or belief, then it's adaptive to suppress that to avoid punishment, while also retaining a desire to still express it when you can get away with it. This then manifests as what could be seen as conflicting sub-agents, with internal conflict and inconsistent behaviour. 6) at the same time there is still genuinely the angle about complexity and unpredictability making it hard to get a complex mind to work coherently and internally aligned. I think that evolution has done a lot of trial and error to set up parameters that result in brain configurations where people end up acting in a relatively sensible way - and even after all that trial and error, lots of people today still end up with serious mental illnesses, failing to achieve almost any of the goals they have (even when the environment isn't stacked against them), dying young due to doing something that they predictably shouldn't have, etc. 7) aligning an AI's sub-agents in a purely simulated environment may not be fully feasible because a lot of the questions that need to be solved are things like "how much priority to allocate to which sub-agent in which situation". E.g. humans come with lots of biological settings that shift the internal balance of sub-agents when hungry, tired, scared,, etc. Some people develop an obsessive focus on a particular topic which may end up being beneficial if they are lucky (obsession on programming that you can turn into a career), or harmful if they are unlucky (an obsession on anything that doesn't earn you money and actively distracts you from it). The optimal prioritization depends on the environment and I don't think there is any theoretically optimal result that would be real-world relevant and that you could calculate beforehand. Rather you just have to do trial-and-error, and while running your AIs in a simulated environment may help a bit, it may not help much if your simulation doesn't sufficiently match the real world. 8) humans are susceptible to internal Goodhart's Law, where they optimize for proxy variables like "sense of control over one's environment", and this also leads them to doing things like playing games or smoking cigarettes to increase their perceived control of the environment without increasing their actual control of the environment. I think that an AI having the same issue is much more likely than it just being able to single-mindedly optimize for a single goal and derive all of its behavior and subgoals from that. Moreover, evolution has put quite a bit of optimization power into developing the right kinds of pr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Auditing failures vs concentrated failures, published by Ryan Greenblatt on December 11, 2023 on The AI Alignment Forum. TL;DR: If you want to notice when powerful AIs are behaving badly, a reasonable strategy is to carefully audit a small fraction of AI actions. This kind of strategy has two kinds of failures which can be tackled somewhat independently: auditing failures, where the audits fail to recognize some bad behavior, and concentrated failures, where the AI causes an irreversible catastrophe in so few actions or such a short time frame that auditing didn't have a chance to review the bad actions. Many oversight techniques are so expensive that using them on every AI action would destroy most of the value of using AIs in the first place: you might want to have humans extensively review all AI-generated code line by line, but this reduces AI usefulness so much that you would often be better off having humans write the code in the first place. A typical way to solve this problem is to just audit actions during an upfront training period (either to directly train an AI, or to train a reward model), but this fails if AIs can confidently tell some deployment inputs from training inputs: they could behave arbitrarily badly on these distinguishable deployment inputs by using a strategy like "if in training or uncertain do X; if in deployment do Y". Here is a better baseline: random audits. In addition to training, apply expensive human auditing to a randomly selected set of deployment actions (e.g. 2% of actions) and check that AIs aren't doing more bad actions than expected. Now for AIs to frequently behave poorly without getting caught, AIs can't just depend on distinguishing training from some deployment inputs: AIs must be able to sneak bad behavior past the auditing process, otherwise, auditing would eventually notice bad actions[1]. Random audits don't solve all issues, there are still: Auditing failures: The auditing process failed to recognize bad behavior[2] (this is the classic scalable oversight problem). Concentrated failures: We failed to handle catastrophic behavior because the catastrophic behavior is concentrated in a small number of actions or a small time frame (these are sometimes called high-stakes failures[3]). This can be further decomposed into two failure modes: Sampling concentrated failure: The auditing process didn't sample the concentrated catastrophic actions. This is catastrophic if AIs can cause a catastrophe in a small number of actions. (For instance, if AIs can cause a catastrophe in 20 bad actions and we only audit 2% of actions, then we would only sample at least one bad action with probability 0.33.). Latency concentrated failure: The auditing process failed to finish reviewing the catastrophic actions before it was too late. This is catastrophic if AIs can coordinate to take enough bad actions in a small amount of time to irreversibly take over. We believe this distinction is often useful when thinking about what different techniques are useful for: sometimes they help you make the expensive auditing better, sometimes they help you audit more the actions that matter the most, and sometimes they help you audit before it's too late.[4] The rest of this post dives deeper into concentrated failures and describes some ways to improve on the random auditing baseline. Sampling concentrated failure AIs could take bad actions rarely, such that they fail a small number of times in expectation throughout deployment and are thus unlikely to behave badly in the small fraction of AI actions which are audited (as seen in our paper). We call this a sampling concentrated failure as the issue is that the AI took bad actions very rarely, which resulted in auditing not sampling bad actions. Latency concentrated failure AIs could coordinate on doing ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Principles For Product Liability (With Application To AI), published by johnswentworth on December 10, 2023 on LessWrong. There were several responses to What I Would Do If I Were Working On AI Governance which focused on the liability section, and had similar criticisms. In particular, I'll focus on this snippet as a good representative: Making cars (or ladders or knives or printing presses or...) "robust to misuse", as you put it, is not the manufacturer's job. The commenter calls manufacturer liability for misuse "an absurd overreach which ignores people's agency in using the products they purchase". Years ago I would have agreed with that; it's an intuitive and natural view, especially for those of us with libertarian tendencies. But today I disagree, and claim that that's basically not the right way to think about product liability, in general. With that motivation in mind: this post lays out some general principles for thinking about product liability, followed by their application to AI. Principle 1: "User Errors" Are Often Design Problems There's this story about an airplane (I think the B-52 originally?) where the levers for the flaps and landing gear were identical and right next to each other. Pilots kept coming in to land, and accidentally retracting the landing gear. Then everyone would be pissed at the pilot for wrecking the bottom of the plane, as it dragged along the runway at speed. The usual Aesop of the story is that this was a design problem with the plane more than a mistake on the pilots' part; the problem was fixed by putting a little rubber wheel on the landing gear lever. If we put two identical levers right next to each other, it's basically inevitable that mistakes will be made; that's bad interface design. More generally: whenever a product will be used by lots of people under lots of conditions, there is an approximately-100% chance that the product will frequently be used by people who are not paying attention, not at their best, and (in many cases) just not very smart to begin with. The only way to prevent foolish mistakes sometimes causing problems, is to design the product to be robust to those mistakes - e.g. adding a little rubber wheel to the lever which retracts the landing gear, so it's robust to pilots who aren't paying attention to that specific thing while landing a plane. Putting the responsibility on users to avoid errors will always, predictably, result in errors. The same also applies to intentional misuse: if a product is widely available, there is an approximately-100% chance that it will be intentionally misused sometimes. Putting the responsibility on users will always, predictably, result in users sometimes doing Bad Things with the product. However, that does not mean that it's always worthwhile to prevent problems. Which brings us to the next principle. Principle 2: Liability Is Not A Ban A toy example: a railroad runs past a farmer's field. Our toy example is in ye olden days of steam trains, so the train tends to belch out smoke and sparks on the way by. That creates a big problem for everyone in the area if and when the farmer's crops catch fire. Nobody wants a giant fire. (I think I got this example from David Friedman's book Law's Order, which I definitely recommend.) Now, one way a legal system could handle the situation would be to ban the trains. One big problem with that approach is: maybe it's actually worth the trade-off to have crop fires sometimes. Trains sure do generate a crapton of economic value. If the rate of fires isn't too high, it may just be worth it to eat the cost, and a ban would prevent that. Liability sidesteps that failure-mode. If the railroad is held liable for the fires, it may still choose to eat that cost. Probably the railroad will end up passing (at least some of) that cost throug...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: PEPFAR, one of the most life-saving global health programs, is at risk, published by salonium on December 10, 2023 on The Effective Altruism Forum. Summary: International funding and coordination to tackle HIV/AIDS and support health systems in lower- and middle-income countries, is at risk of not being renewed by US Congress, due to demands that it should be linked to new abortion-related restrictions in recipient countries. This program is estimated to have saved over 20 million lives since it was launched by the Bush Administration in 2003, and even now averts over a million HIV/AIDS deaths annually. Since it has also helped support health systems in LMICs, and tackle malaria and tuberculosis, its impact is likely greater than this. In my view this is the most important risk to global health we face today, and I think it isn't getting enough attention. If anyone is interested in research, writing or advocacy on this issue, please do so. If you are interested in jointly working on this, or if you already know of ongoing efforts, please comment below or get in touch. My email: saloni@ourworldindata.org Relevant background reading: The U.S. President's Emergency Plan for AIDS Relief (PEPFAR), the largest commitment in history by any single country to address a disease, is estimated to have averted 25 million deaths from AIDS and enabled 5.5 million babies to be born free from HIV infection over the past 20 years.1 It has provided more than $100 billion in funding for HIV prevention, care, and treatment internationally, supporting 55 low- and middle-income countries that are collectively home to 78% of all people living with HIV. Together with the Global Fund to Fight AIDS, Tuberculosis, and Malaria, PEPFAR has transformed AIDS in low-income countries, especially those in Africa, from a death sentence to a readily treatable chronic disease by deploying programs that provide antiretroviral treatment even in the most remote villages. Right from the start, PEPFAR was more than just an AIDS program; it partnered with countries in Africa to support the development of health systems for essential community services, trained thousands of health care workers, fostered security and stability in affected countries, and engendered hope amid a devastating global AIDS crisis. Karim et al. (2023) Why is it at risk? Republican colleagues [...] accuse the Biden administration of using PEPFAR to fund abortion providers overseas and House Democrats who refuse to reinstate Trump administration rules that prohibited foreign aid going to groups that provide or counsel on abortions. Discussions about a compromise that would extend the program for more than one year but less than five, with language stressing the existing ban on federal money directly paying for abortions, have collapsed. Now, the best hope for re-upping the $7 billion annual program is a government spending process beset by delays and divisions and slated to drag into January and February with no guarantee of success. PEPFAR can hobble along without reauthorization unless there's a prolonged government shutdown. But its backers say that without a long-term U.S. commitment, groups fighting HIV and AIDS around the world will struggle to hire staff and launch long-term projects. Complicating any hope for compromise is the 2024 election. Congress passed two short-term funding patches that expire in January and February. That eliminated the possibility of the typical end-of-year omnibus bill that many on both sides of the PEPFAR fight saw as the best vehicle for its reauthorization and kicked the fight into an election year when compromise - particularly on a contentious issue like abortion - will be more challenging. Politico [7 Dec. 2023] The lawmakers stalling the reauthorization are seeking to impose on PEPFAR a prohibition...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I spent a lot of time and money on one dog. Do I need help?, published by tobiasleenaert on December 10, 2023 on The Effective Altruism Forum. This is about a personal experience - rescuing a dog on a trip in Mexico - that helped me realize how I wrestle with being effective. My girlfriend and me were recently in Mexico. After speaking at a conference, we took two weeks off in Yucatan. We had both been aware that we'd meet a lot of stray animals. We knew that this situation could potentially spoil our vacation - which was meant as a break from our daily involvement in animal activism (and thus animal suffering). So as well as we could, we avoided getting too close to any animal we saw in the streets. That worked, until it didn't. Near the end of our trip, I ran into a quite unhealthy looking dog who was riddled with ticks. We spent half an hour taking the ticks out and by the time we were done with him, we knew we wouldn't let him lie there. I called a few shelters, asking if they could take the dog in, but they were all full. Contacting local activists, we miraculously found a place nearby where he'd be able to stay indefinitely. We brought him there right away, leaving him in a pen (there were other dogs that needed to get used to him). When we went away, it was with a bad feeling. Later that night, we decided we would pick him up again the next morning (our hotel didn't allow dogs so we had to wait) and would find another place for him. When we went back the next day, we were told that the dog had escaped. We saw where he had bit through the fence, probably in desperation and fear of the barking of the other dogs. We felt devastated, thinking that instead of helping him, we had put his life at risk. There was a very busy road right next to the property, and the place where we had found him - his home turf - was six kilometers away. By now we had bonded with this dog - which we called Tlalok, after the Aztec rain god - and mourned for the rest of the day, as if one of our own dogs had just died. We actually had difficulty understanding why this whole situation affected us so much. Maybe it was because Tlalok was such an incredibly friendly, trusting - and at the same time needy - being. The next day, one day before we'd fly home, we decided we wouldn't give up on him. I made a Spanish "LOST" flyer that we copied in our hotel and then distributed in the village where we'd found him. The flyer promised a reward of 5000 pesos (250 dollars). We spoke to many people online and offline, contacted vets, shelters and activists… At one point that day, a vet we were in touch with sent us the picture of a dog that had walked into a hotel where coincidentally a friend of hers was staying. It was Tlalok! We rushed to the village for the second time that day and half an hour later, we felt the relief as we hugged him. We brought him to our hotel and the next night, to a shelter, which was full but agreed to take him in and make sure he got all the necessary care, but only if we agreed to adopt him. We said yes, not having any alternative solution. We'd figure out later what exactly we would do with Tlalok, but the plan was set in motion to have him fly over to Belgium, where we live. Four weeks later, we picked Tlalok up at Schiphol Airport near Amsterdam - where he arrived a day later than planned so that we had to stay an extra night there. As I'm writing this, he's in our kitchen with our four other dogs. We're looking to have him adopted, knowing that with each day that he's here, it will be harder to part ways. In the meantime, we also remained in contact with the vet in Mexico and paid her a couple of hundred dollars to spay/neuter Tlalok's siblings and mother. *** After reading some of this story, which I had posted - with some pictures - on Facebook, an EA friend told me...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How LDT helps reduce the AI arms race, published by Tamsin Leake on December 10, 2023 on The AI Alignment Forum. (Epistemic status: I think this is right?) Alice is the CEO of ArmaggAI, and Bob is the CEO of BigModelsAI, two major AI capabilities organizations. They're racing to be the first to build a superintelligence aligned to their respective CEV which would take over the universe and satisfy their values. Alice would prefer to slow down so she has a lot more time to make sure that what her company is building will be aligned; but she's worried about Bob's company taking advantage of that and racing themselves, resulting in Bob's utility function being {what the lightcone is filled with} instead of Alice's; so she feels like she has no choice but to race, to maximize her own utility. Likewise vice-versa. This state of affairs is far from the pareto frontier of {their utility functions, each weighed by how likely they are to be the one to build an aligned superintelligence}: it causes a lot more worlds where everyone is dead, instead of either Alice's or Bob's utility function being maximized. But they don't really get all this LDT business; it seems complicated to them. If only they had a convenient lesswrong post spelling out exactly what they should do, in clear easy-to-follow terms! The procedure they should implement; Alice should: In worlds where she does launch a superintelligence aligned to itself, and it takes over the world and builds an Alice::utopia, she should tell her AI: Actually, don't just maximize my own utility function. Maximize a (threat-resistant) mixture of my utility function and Bob's, weighed by what ratio of timelines you think we each launched an aligned superintelligence in, but also weighed by the probability that both me and Bob implement this procedure. Before that, she should slow down their company's capabilities progress. And likewise for bob. Now, their respective aligned superintelligences aren't omniscient; it could be that they over-estimate or under-estimate how likely Bob or Alice would've been to implement this procedure, for example. But one thing's pretty much for sure: neither Alice nor Bob are able to fool the future-superintelligence into thinking that they'd implement this procedure, when actually they wouldn't. Because it's superintelligent. If Alice wins, she knows Bob will follow the procedure because her superintelligence can tell (better than Bob can fake). And Bob doesn't have to count on wishful thinking to know that Alice would indeed do this instead of defecting, because in worlds where he wins, he can his superintelligence if Alice would implement this procedure. They're each kept in check by the other's future-winning-self, and they can each rely on this being superintelligently-checked by their respective future selves. So the only way Alice has to get some of her utility maximized in worlds where Bob wins, is to actually behave like this, including before either has launched a superintelligence. And likewise for Bob. Their incentive gradient is in the direction of being more likely to follow this procedure, including slowing down their capabilities progress - and thus decreasing the amount of worlds where their AI is unaligned and everyone dies forever. In the real world, there are still Bob's and Alice's who don't implement this procedure, but that's mostly because they don't know/understand that if they did, they would gain more utility. In many cases, it should suffice for them to be informed that this is indeed where their utility lies. Once someone has demonstrated that they understand how LDT applies here, and that they're generally rational, then they should understand that implementing this protocol (including slowing down AI capabilities) is what maximizes their utility, and so you can count on...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: PhD on Moral Progress - Bibliography Review, published by Rafael Ruiz on December 10, 2023 on The Effective Altruism Forum. Epistemic Status: I've researched this broad topic for a couple of years. I've read about 30+ books and 100+ articles on the topic so far (I'm not really keeping count). I've also read many other works in the related areas of normative moral philosophy, moral psychology, moral epistemology, moral methodology, and metaethics, since it's basically my area of specialization within philosophy. This project will be my PhD thesis. However, I still have 3 years of the PhD to go, so a substantial amount of my opinions on the matter are subject to changes. Disclaimer: I have received some funding as a Forethought Foundation Fellow in support of my PhD research. But all the opinions expressed here are my own. Index. Part I - Bibliography Review Part II - Preliminary Takes and Opinions (I'm writing it, coming very soon!) More parts to be published later on. Introduction. Hi everyone, this is my first proper research-related post on the EA Forum, on a topic that I've been working on for several years, since even before my PhD, and now as part of my PhD in Philosophy at the London School of Economics. This post is the start of a series on my work on the topic of Moral Progress, which includes and intersects with Moral Circle Expansion (also called Inclusivism or Moral Inclusion), Moral Progress, Social Progress, Social Movements, the mechanisms that drive progress and regress, the possibilities of measuring these phenomena, and policy or practical implications. This first post is a bibliography review, which I hope will serve to orient future researchers that might want to tackle the same or similar topics. Hopefully it will help them to save time by separating the wheat from the chaff, the good research articles and books from the minor contributions. Initially, I had my reservations about doing a Bibliography Review, since now we have GPT4 which is quite good at purely neutral descriptive summarizing, so I felt maybe perhaps this work wasn't needed. However, given that now we have it as a good research assistant for pure facts, that also allows me more freedom to be more opinionated in my bibliography review. I'll try to tell you what I think is super worth reading, and what is "meh, skim it if you have free time", so you can sift through the literature in a more time-efficient way. The eventual goal outcome of the whole project would be to distil the main insights into book on the topic of Moral Progress with serious contributions to the current literature within interdisciplinary moral philosophy, but that probably won't happen until I finish my PhD thesis manuscript around 2026. Then after that, I'll have to rewrite that manuscript to turn it into a more accessible book, so it probably wouldn't be published until a later date. I'm also not sure just yet whether it would be an academic book on a University Press or something closer to What We Owe The Future, which aims to be accessible for a broader audience. So the finished work is quite a long way. On the brighter side, I will publish some of the key findings and takeaways on the EA Forum, probably in summarized form rather than the excruciatingly slow pace of writing in philosophy, which often takes 20 pages to make some minor points. Instead of that, I guess I'll post something closer to digestible bullet points with my views, attempting to foster online discussion, and then defend them in more detail over time and in the eventual book. Your feedback will of course be appreciated, particularly if I change my mind on substantial issues, connect me with other researchers, etc. So let's put our collective brains together (this is a pun having to do with cultural evolution that you might not understand y...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Unpicking Extinction, published by ukc10014 on December 10, 2023 on LessWrong. TL;DR Human extinction is trending: there has been a lot of noise, mainly on X, about the apparent complacency amongst e/acc with respect to human extinction. Extinction also feels adjacent to another view (not particular to e/acc) that 'the next step in human evolution is {AI/AGI/ASI}'. Many have pushed back robustly against the former, while the latter doesn't seem very fleshed out. I thought it useful to, briefly, gather the various positions and summarise them, hopefully not too inaccurately, and perhaps pull out some points of convergence. This is a starting point for my own research (on de-facto extinction via evolution). There is nothing particularly new in here: see the substantial literature in the usual fora for instance. Thomas Moynihan's X-risk (2020) documents the history of humanity's collective realisation of civilisational fragility, while Émile P. Torres' works (discussed below) set out a possible framework for an ethics of extinction. My bottom line is: a) the degree of badness (or goodness) of human extinction seems less obvious or self-evident than one might assume, b) what we leave behind if and when we go extinct matters, c) the timing of when this happens is important, as is d) the manner in which the last human generations live (and die). Relevant to the seeming e/acc take (i.e. being pretty relaxed about possible human extinction): it seems clear that our default position (subject to some caveats) should be to delay extinction on the grounds that a) it is irreversible (by definition), and b) so as to maximise our option value over the future. In any case, the e/acc view, which seems based on something (not very articulate) something entropy crossed with a taste for unfettered capitalism, is hard to take seriously and might even fail on its own terms. Varieties of extinction The Yudkowsky position (My take on) Eliezer's view is that he fears a misaligned AI (not necessarily superintelligent), acting largely on its own (e.g. goal-formation, planning, actually effecting things in the world), eliminates humans and perhaps all life on Earth. This would be bad, not just for the eliminated humans or their descendants, but also for the universe-at-large in the sense that intelligently-created complexity (of the type that humans generate) is an intrinsic good that requires no further justification. The vast majority of AI designs that Eliezer foresees would, through various chains of events, result in a universe with much less of these intrinsic goods. He spells it out here in the current e/acc context, and clarifies that his view doesn't hinge on the preservation of biological humans (this was useful to know). He has written copiously and aphoristically on this topic, for instance Value is Fragile and the Fun Theory sequence. The Bostrom variant Nick Bostrom's views on human extinction seem to take a more-happy-lives-are-better starting point. My possibly mistaken impression is that, like Eliezer, he seems to value things like art, creativity, love, in the specific sense that a future where they didn't exist would be a much worse one from a cosmic or species-neutral perspective. He describes an 'uninhabited society' that is technologically advanced and builds complex structures, but that 'nevertheless lacks any type of being that is conscious or whose welfare has moral significance' (Chapter 11, p. 173 of Superintelligence (2014)). To my knowledge, he doesn't unpick what precisely about the uninhabited society would actually be bad and for whom (possibly this is well-understood point or a non-question in philosophy, but I'm not sure that is the case, at least judging from (see below) Benatar, Torres, this paper by James Lenman, or for that matter Schopenhauer). A more tang...