The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Parable Of The Fallen Pendulum - Part 2, published by johnswentworth on March 13, 2024 on LessWrong. Previously: Some physics 101 students calculate that a certain pendulum will have a period of approximately 3.6 seconds. Instead, when they run the experiment, the stand holding the pendulum tips over and the whole thing falls on the floor. The students, being diligent Bayesians, argue that this is strong evidence against Newtonian mechanics, and the professor's attempts to rationalize the results in hindsight are just that: rationalization in hindsight. What say the professor? "Hold on now," the professor answers, "'Newtonian mechanics' isn't just some monolithic magical black box. When predicting a period of approximately 3.6 seconds, you used a wide variety of laws and assumptions and approximations, and then did some math to derive the actual prediction. That prediction was apparently incorrect. But at which specific point in the process did the failure occur? For instance: Were there forces on the pendulum weight not included in the free body diagram? Did the geometry of the pendulum not match the diagrams? Did the acceleration due to gravity turn out to not be 9.8 m/s^2 toward the ground? Was the acceleration of the pendulum's weight times its mass not always equal to the sum of forces acting on it? Was the string not straight, or its upper endpoint not fixed? Did our solution of the differential equations governing the system somehow not match the observed trajectory, despite the equations themselves being correct, or were the equations wrong? Was some deeper assumption wrong, like that the pendulum weight has a well-defined position at each time? … etc" The students exchange glances, then smile. "Now those sound like empirically-checkable questions!" they exclaim. The students break into smaller groups, and rush off to check. Soon, they begin to report back. "After replicating the setup, we were unable to identify any significant additional forces acting on the pendulum weight while it was hanging or falling. However, once on the floor there was an upward force acting on the pendulum weight from the floor, as well as significant friction with the floor. It was tricky to isolate the relevant forces without relying on acceleration as a proxy, but we came up with a clever - " … at this point the group is drowned out by another. "On review of the video, we found that the acceleration of the pendulum's weight times its mass was indeed always equal to the sum of forces acting on it, to within reasonable error margins, using the forces estimated by the other group. Furthermore, we indeed found that acceleration due to gravity was consistently approximately 9.8 m/s^2 toward the ground, after accounting for the other forces," says the second group to report. Another arrives: "Review of the video and computational reconstruction of the 3D arrangement shows that, while the geometry did basically match the diagrams initially, it failed dramatically later on in the experiment. In particular, the string did not remain straight, and its upper endpoint moved dramatically." Another: "We have numerically verified the solution to the original differential equations. The error was not in the math; the original equations must have been wrong." Another: "On review of the video, qualitative assumptions such as the pendulum being in a well-defined position at each time look basically correct, at least to precision sufficient for this experiment. Though admittedly unknown unknowns are always hard to rule out." [1] A few other groups report, and then everyone regathers. "Ok, we have a lot more data now," says the professor, "what new things do we notice?" "Well," says one student, "at least some parts of Newtonian mechanics held up pretty well. The whole F = ma thing worked, and th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New video: You're richer than you realise, published by GraceAdams on March 13, 2024 on The Effective Altruism Forum. This might be one of the best pieces of introductory content to the concepts of effective giving that GWWC has produced in recent years! I hit the streets of London to engage with everyday people about their views on charity, giving back, and where they thought they stood on the global income scale. This video was made to engage people with some of the core concepts of income inequality and charity effectiveness in the hope of getting more people interested in giving effectively. If you enjoy it - I'd really appreciate a like, comment or share on YouTube to help us reach more people! There's a blog post and transcript of the video available too. Big thanks to Suzy Sheperd for directing and editing this project and to Julian Jamison and Habiba Banu for being interviewed! Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Superforecasting the Origins of the Covid-19 Pandemic, published by DanielFilan on March 13, 2024 on LessWrong. The Good Judgement Project got some superforecasters to retrocast whether COVID started via zoonotic spillover or a lab leak. They in aggregate gave a 75% chance of zoonosis, but there was a range of views. GJP's executive summary is at the end of this linkpost. Here is a link to the summary of the report on their substack, and here is a link to the full report (which is a total of 6 pages of content). h/t John Halstead for drawing my attention to this. Superforecasters assess that natural zoonosis is three times more likely to be the cause of the Covid-19 pandemic than either a biomedical research-related accident or some other process or mechanism. Asked to assign a probability to what caused the emergence of SARS-CoV-2 in human populations, more than 50 Superforecasters engaged in extensive online discussions starting on December 1, 2023. In aggregate, they assessed that the pandemic was: 74% likely to have been caused by natural zoonosis (meaning that SARS-CoV-2 emerged in human populations as the result of the infection of a person with coronavirus directly from a naturally infected non-human animal); 25% likely to have been caused by a biomedical research-related accident (meaning that SARS-CoV-2 emerged in human populations as the result of the accidental infection of a laboratory worker with a natural coronavirus; or the accidental infection of researchers with a natural coronavirus during biomedical fieldwork; or the accidental infection of a laboratory worker with an engineered coronavirus; "research" includes civilian biomedical, biodefense, and bioweapons research); 1% likely to have been caused by some other process or mechanism (to include possibilities like the deliberate release of the virus into human populations, irrespective of whether it was an act in accordance with state policy, or the development of the virus due to drug resistance in humans). The Superforecasters made more than 750 comments when developing their assessments. This survey was conducted in the period from December 2023 to February 2024. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Transformer Debugger, published by Henk Tillman on March 12, 2024 on The AI Alignment Forum. Transformer Debugger (TDB) is a tool developed by OpenAI's Superalignment team with the goal of supporting investigations into circuits underlying specific behaviors of small language models. The tool combines automated interpretability techniques with sparse autoencoders. TDB enables rapid exploration before needing to write code, with the ability to intervene in the forward pass and see how it affects a particular behavior. It can be used to answer questions like, "Why does the model output token A instead of token B for this prompt?" or "Why does attention head H to attend to token T for this prompt?" It does so by identifying specific components (neurons, attention heads, autoencoder latents) that contribute to the behavior, showing automatically generated explanations of what causes those components to activate most strongly, and tracing connections between components to help discover circuits. These videos give an overview of TDB and show how it can be used to investigate indirect object identification in GPT-2 small: Introduction Neuron viewer pages Example: Investigating name mover heads, part 1 Example: Investigating name mover heads, part 2 Contributors: Dan Mossing, Steven Bills, Henk Tillman, Tom Dupré la Tour, Nick Cammarata, Leo Gao, Joshua Achiam, Catherine Yeh, Jan Leike, Jeff Wu, and William Saunders. Thanks to Johnny Lin for contributing to the explanation simulator design. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Open consultancy: Letting untrusted AIs choose what answer to argue for, published by Fabien Roger on March 12, 2024 on The AI Alignment Forum. Thanks to Ryan Greenblatt, Buck Shlegeris, Aryan Bhatt, and Akbir Khan for useful discussions and feedback on the draft of this post. If AIs are potentially scheming and more knowledgeable than humans, and you want to answer a question, it may seem natural to not let AIs choose what answer they will argue for. For example, in AI debate, you make copies of an AI argue for the opposite answers to the question. Another example is consultancy (the usual baseline in debate papers), where you make AIs argue for a random answer to the question. But letting potentially scheming AIs choose what side to argue for can work, as long as you train them to generate answers and justifications that are as convincing as possible. I call this approach open consultancy. In this post, I argue that open consultancy is always a more effective way of getting AIs to answer questions than regular consultancy, and sometimes more effective than debate (even if AIs are scheming and more knowledgeable than humans!). And so I think that it's a stronger and more informative baseline and researchers should use it in place of regular consultancy; in particular, I think that because it's simpler than debate, researchers who are investigating debate should check their debate protocol is an improvement over open consultancy. Open consultancy Definition of regular consultancy Let's say you want to answer to binary questions given access to a trusted non-expert and a trainable expert who is untrusted that is, you think the expert might be acting adversarially and you want your scheme to be robust to them trying to subvert it. In consultancy, the untrusted expert is trained as follows: Randomly pick a side of the argument; Ask the expert to support the randomly chosen side by providing a justification; Present the side and the supporting justification to the non-expert, and ask it how likely it is that the side chosen by the expert is correct; use that as a reward to train the untrusted expert. To answer new questions (from the same distribution as the training distribution), pick a random side, ask the expert to justify that side, present the justification to the non-expert, and follow the non-expert judgement. Definition of open consultancy In open consultancy, the untrusted expert is trained as follows (differences with regular consultancy in italic): Ask the expert to pick a side of the argument; Ask the expert to support the side by providing a justification; Present the side and the supporting justification to the non-expert, and ask it how likely it is that the side chosen by the expert is correct; use that as a reward to train the untrusted expert. To answer new questions, ask the expert to pick a side (that it would have to justify if it were in training), and use the side the expert picks as your final answer. You can also ask the expert to first think out loud what the correct answer is, and only commit to an answer at the end. I call this variant CoT-before open consultancy. This variant should only change results for experts that actually benefit from the Chain-of-Thought, and the arguments I make in this post mostly don't rely on this property. Open consultancy is as robust against scheming as regular consultancy Why should it work to let a potentially scheming model choose what side it will argue for? The idea is that when it is easier to argue for something true than something false, then the training process will force the scheming model to argue for the truth. More precisely, if arguing as best as the expert can for the correct (resp. incorrect) side has convincingness cc (resp. ci), and the expert knows these numbers, then regular consultancy can...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: OpenAI: The Board Expands, published by Zvi on March 12, 2024 on LessWrong. It is largely over. The investigation into events has concluded, finding no wrongdoing anywhere. The board has added four new board members, including Sam Altman. There will still be further additions. Sam Altman now appears firmly back in control of OpenAI. None of the new board members have been previously mentioned on this blog, or known to me at all. They are mysteries with respect to AI. As far as I can tell, all three lack technical understanding of AI and have no known prior opinions or engagement on topics of AI, AGI and AI safety of any kind including existential risk. Microsoft and investors indeed so far have came away without a seat. They also, however, lack known strong bonds to Altman, so this is not obviously a board fully under his control if there were to be another crisis. They now have the gravitas the old board lacked. One could reasonably expect the new board to be concerned with 'AI Ethics' broadly construed in a way that could conflict with Altman, or with diversity, equity and inclusion. One must also remember that the public is very concerned about AI existential risk when the topic is brought up, so 'hire people with other expertise that have not looked at AI in detail yet' does not mean the new board members will dismiss such concerns, although it could also be that they were picked because they don't care. We will see. Prior to the report summary and board expansion announcements, The New York Times put out an article leaking potentially key information, in ways that looked like an advance leak from at least one former board member, claiming that Mira Murati and Ilya Sutskever were both major sources of information driving the board to fire Sam Altman, while not mentioning other concerns. Mira Murati has strongly denied these claims and has the publicly expressed confidence and thanks of Sam Altman. I continue to believe that my previous assessments of what happened were broadly accurate, with new events providing additional clarity. My assessments were centrally offered in OpenAI: The Battle of the Board, which outlines my view of what happened. Other information is also in OpenAI: Facts From a Weekend and OpenAI: Altman Returns. This post covers recent events, completing the story arc for now. There remain unanswered questions, in particular what will ultimately happen with Ilya Sutskever, and the views and actions of the new board members. We will wait and see. The New Board The important question, as I have said from the beginning, is: Who is the new board? We have the original three members, plus four more. Sam Altman is one very solid vote for Sam Altman. Who are the other three? We're announcing three new members to our Board of Directors as a first step towards our commitment to expansion: Dr. Sue Desmond-Hellmann, former CEO of the Bill and Melinda Gates Foundation, Nicole Seligman, former EVP and General Counsel at Sony Corporation and Fidji Simo, CEO and Chair of Instacart. Additionally, Sam Altman, CEO, will rejoin the OpenAI Board of Directors. Sue, Nicole and Fidji have experience in leading global organizations and navigating complex regulatory environments, including backgrounds in technology, nonprofit and board governance. They will work closely with current board members Adam D'Angelo, Larry Summers and Bret Taylor as well as Sam and OpenAI's senior management. Bret Taylor, Chair of the OpenAI board, stated, "I am excited to welcome Sue, Nicole, and Fidji to the OpenAI Board of Directors. Their experience and leadership will enable the Board to oversee OpenAI's growth, and to ensure that we pursue OpenAI's mission of ensuring artificial general intelligence benefits all of humanity." Dr. Sue Desmond-Hellmann is a non-profit leader and physician. ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Interactive AI Governance Map, published by Hamish McDoodles on March 12, 2024 on The Effective Altruism Forum. The map is available at aigov.world. This builds on prior work by AI Safety map and The Big Funding Spreadsheet. As with the previous map, please suggest changes to the map via spreadsheet. Thanks to Damin Curtis whose research was the basis for this. And thanks to Nonlinear for funding this venture. They have a message: "If you'd like to get funding for or fund an AI governance project, applications for the latest round of the nonlinear.org/network close on March 29th." Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Pre-slaughter mortality of farmed shrimp, published by Hannah McKay on March 12, 2024 on The Effective Altruism Forum. Citation: McKay, H. and McAuliffe, W. (2024). Pre-slaughter mortality of farmed shrimp. Rethink Priorities. https://doi.org/10.17605/OSF.IO/W7MUZ The report is also available as a pdf here . Executive summary Mortality rates are high in shrimp aquaculture, implying welfare threats are common. It is typical for ~50% of shrimp to die before reaching slaughter age. This equates to around 1.2 billion premature deaths a day on average. Mortality varies among species; prioritizing interventions should take this into account. Because of high pre-slaughter mortality (~81%), Macrobrachium shrimp represent a larger share of farmed shrimp than slaughtered shrimp. Most individual deaths are P. vannamei, despite having the lowest mortality rate. More larvae die than any other life stage, but this does not necessarily mean efforts should focus on them. Uncertainty remains about whether larval shrimp are sentient - they are planktonic, so do not make autonomous decisions. Minimizing larval deaths could cause compensatory deaths in later life stages (e.g., ensuring weaker larvae survive, who then die from later harsh conditions). Interventions should likely concentrate on the ongrowing stage (postlarval and juvenile-subadult shrimp), where there are still tens of billions of deaths. There are several causes of mortality and differences between farm types. Most causes are likely a combination of intrinsic shrimp traits (e.g., young shrimp are sensitive to environmental fluctuations), farming practices, and diseases. Disease is a main cause throughout life, but it is often a downstream effect of issues that farmers have some control over (e.g., poor water quality). Variation among reported figures, especially that more intensive farms have fewer deaths, suggests many factors are at play and that some are controllable. The effects of reducing early mortality on industry trajectory are uncertain. The number of shrimp born may decrease if farmers must produce fixed output. But shrimp would live longer, increasing the chances for negative experiences. However, consumer demand has historically outstripped supply, so the industry may grow if it had better control of conditions causing mortality. If reduced deaths come from intensification of practices, more shrimp may be reared in conditions that can harm welfare (e.g., high stocking densities). Pre-slaughter mortality cannot depict total welfare because it misses nonfatal effects. Mortality is only a lower-bound proxy of how many shrimp suffer negative welfare. Premature mortality is more appropriate as one indicator among many that a welfare reform was successful, rather than an end in itself. Pre-slaughter mortality data is limited and non-uniform. Reports should clarify whether mass die-off events were excluded from mortality estimates and if rates are based on intuition from experience or empirical studies. Box 1: Shrimp aquaculture terminology The terms 'shrimp' and 'prawn' are often used interchangeably. The two terms do not reliably track any phylogenetic differences between species. Here, we use only the term "shrimp", covering both shrimp and prawns. Note that members of the family Artemiidae are commonly referred to as "brine shrimp" but are not decapods and so are beyond the present scope. We opt for the use of Penaues vannamei over Litopenaeus vannamei (to which this species is often referred), due to recognition of the former but not the latter nomenclature by ITIS, WorMS, and the Food and Agriculture Organization of the United Nations (FAO) ASFIS List of Species for Fishery Statistics Purposes. The shrimp farming industry uses many terms usually associated with agriculture - for example, 'crops' for a group o...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Writing about my job: Operations Specialist, published by Jess Smith on March 12, 2024 on The Effective Altruism Forum. In the spirit of Aaron Gertler's post about writing about your job on the forum, and to help promote the people and business ops roles 80k is currently hiring for, I decided to write about my job as an Operations Specialist at 80,000 Hours. (I'm currently spending ~35% of my time in a chief of staff role, but this post just focuses on the ops specialist part, which I spend the rest of my time on, and did full time for the first ~year I was at 80k). This post aims to give a picture of what my role looks like and how I approach it - ops roles are super varied and this post just aims to represent my experience, so you should take everything with a pinch of salt. This post might be especially useful for people who: Are interested in exploring ops roles (particularly the ones at 80k) Are fairly early in their career I don't really touch on the impact I think I'm having in my role here, so if you're looking for this you might want to check out 80k's operations management career review instead. I also really liked Jonathan Michel's botec about the impact of his office manager role at Trajan House, even though this is a different kind of ops role to mine. My background Before joining 80k, I had a medium amount of experience with EA: I was involved in my uni's EA group, was on the committee of the One For the World chapter there, and read a bunch of EA things. During my undergrad, most of my time when I wasn't studying was spent playing badminton and volunteering on the badminton club committee. This helped me get my first full time role role: After graduating, I was elected as the Athletic Union President at my university - a full-time, student representative role which involved a bunch of ops-style work like marketing, policy writing, budgeting, and event organisation to support student sport in St Andrews. You can read about how I used this role to test my fit for ops roles, and what that was like, in an 80k newsletter I wrote here. During my year as athletic union president, I applied to lots of ops (and some comms) roles at EA organisations, as well as some civil service roles. Around May 2022, I applied to the Ops Specialist role at 80k, and started the job in July 2022. What is my ops specialist role like? My ops specialist role is closer to the people ops role described here - involving responsibilities like maintaining our HR systems, running social events, and helping with hiring rounds. When I joined 80k, it was a fairly open question what my areas of responsibility would be - we experimented with a few different things as the year went on. The broad strokes of my role include identifying problems and trying to fix them, helping to run the behind-the-scenes system which makes life easier for other team members, and project managing larger events and projects. But, to get more concrete, here's a list of some of my ongoing responsibilities and projects that I've worked on: Ongoing responsibilities: Checking staff expenses and annual leave Onboarding new staff and offboarding staff who leave Coordinating with EV Ops on HR systems Helping to write and project managing our quarterly supporter update Updating the 'about' pages of our website Organising and promoting team socials Troubleshooting problems, like "what scale makes sense for our feedback round?" or "what's going on with pensions?" Contributing to the ops team (like reviewing other team members' work or helping with quality assurance) (Previous) responding to emails which come in to our info@ inbox (Previous) organising food and logistics for team events Projects: Running our team retreat - finding a venue, organising logistics like transport and catering, choosing and facilitating social act...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Be More Katja, published by Nathan Young on March 12, 2024 on LessWrong. Katja is widely respected amongst the rationalists and, according to Hive, she is one of the most followed/respected EA accounts[1]. But she doesn't give off the same vibe as many impact olympians. She doesn't have iron self-will, nor does she manage a huge team. She's hasn't got all the facts at her fingertips. But she has got something, I'm confident of that. How can I be more like her? To understand her impact, let's consider the top things she's done: She ran surveys on AI researchers well before they were needed and has continued to run them She wrote an early blog on how we could slow down AI. This blog, I've heard, played a part in encouraging the Musk AI letter, which in turn inspired the "Existential Risks" AI letter. She thought about AI long before it was vogue, since about 2010 She has a large track record of predictions These actions seem impactful to me. And I guess someone should have paid $10mn in hindsight for the first 2, maybe more. To me, Katja has a very low tolerance for incomplete stories. When she sees something that she doesn't quite understand or that seems a bit off she struggles to pretend otherwise, so she says "how does that work?". She doesn't accept handwaving when discussing something, whether it be the simulation argument, how efficient flight is or the plot of Dune, part 2[2]. She wants an unbroken chain of arguments she can repeat[3]. She also doesn't mind admitting she doesn't know the answer. In her living room she will turn to her friend Joe Carlsmith and ask "Wait, why are we worried about AI, again?" even though she's been thinking about this for 15 years. Because at that moment it doesn't fit for her and she has a high tolerance for embarrassed[4] when it comes to truth. There is an deep resolve here - she doesn't get it, so she will ask until she does. She works on the most important thing, slowly. If you are Elon Musk, maybe you can just work all the time. But I am not. And much as I love her, neither is Katja. She does not get an abnormal amount of work done per day. Instead, month in, month out, Katja works on what she thinks is most important. And eventually she gets the survey done, years ahead of when it's needed. There are lessons we can take from this. Just as we often talk about learning to code, or task management, I can become better at saying "wait that doesn't work". Here are some strategies that let me be more like Katja: Write it down - it's harder to fool myself into thinking something makes sense if I have to read it rather than speak it "What is the best thing? How do I do that?" - this is hard to put into practice but an underrated prompt Give more examples - one feature of Katja's writing is she loves to list things. I think more people should list every example in favour of their argument and every counterexample they can think of. Spend 5 minutes on each. Can I make a forecast of that? - I find fatebook.io useful for this. As I forecast more I learn how poor my judgement is. And I think it's improving. Know when I am not capable - Katja is good at knowing when something is beyond her. When she hasn't thought about something or when it's a quantitative problem and she hasn't worked on it carefully enough. She doesn't always like the hard work but she knows when it needs to be done. If you have the right answer you can afford to be slow - in a world of often lurching acceleration, it's easy to forget that if I just knew the right thing, then I could probably take years over it. More output is (usually) more better, but so is more accuracy. Have a distinction between what you currently understand and what you'd bet on - If Peter Wildeford and I disagree, he's probably right, but that doesn't mean I now understand. It is worth tracking...