Astral Codex Ten Podcast

https://astralcodexten.substack.com/p/yudkowsky-contra-christiano-on-ai Previously in series: Yudkowsky Contra Ngo On Agents, Yudkowsky Contra Cotra On Biological Anchors Prelude: Yudkowsky Contra Hanson In 2008, thousands of blog readers - including yours truly, who had discovered the rationality community just a few months before - watched Robin Hanson debate Eliezer Yudkowsky on the future of AI. Robin thought the AI revolution would be a gradual affair, like the Agricultural or Industrial Revolutions. Various people invent and improve various technologies over the course of decades or centuries. Each new technology provides another jumping-off point for people to use when inventing other technologies: mechanical gears → steam engine → railroad and so on. Over the course of a few decades, you've invented lots of stuff and the world is changed, but there's no single moment when "industrialization happened". Eliezer thought it would be lightning-fast. Once researchers started building human-like AIs, some combination of adding more compute, and the new capabilities provided by the AIs themselves, would quickly catapult AI to unimaginably superintelligent levels. The whole process could take between a few hours and a few years, depending on what point you measured from, but it wouldn't take decades. You can imagine the graph above as being GDP over time, except that Eliezer thinks AI will probably destroy the world, which might be bad for GDP in some sense. If you come up with some way to measure (in dollars) whatever kind of crazy technologies AIs create for their own purposes after wiping out humanity, then the GDP framing will probably work fine. For transhumanists, this debate has a kind of iconic status, like Lincoln-Douglas or the Scopes Trial. But Robin's ideas seem a bit weird now (they also seemed a bit weird in 2008) - he thinks AIs will start out as uploaded human brains, and even wrote an amazing science-fiction-esque book of predictions about exactly how that would work. Since machine learning has progressed a lot faster than brain uploading has, this is looking less likely and probably makes his position less relevant than in 2008. The gradualist torch has passed to Paul Christiano, who wrote a 2018 post Takeoff Speeds revisiting some of Hanson's old arguments and adding new ones. (I didn't realize this until talking to Paul, but "holder of the gradualist torch" is a relative position - Paul still thinks there's about a 1/3 chance of a fast takeoff.) Around the end of last year, Paul and Eliezer had a complicated, protracted, and indirect debate, culminating in a few hours on the same Discord channel. Although the real story is scattered over several blog posts and chat logs, I'm going to summarize it as if it all happened at once. Gradatim Ferociter Paul sums up his half of the debate as: There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles. (Similarly, we'll see an 8 year doubling before a 2 year doubling, etc.) That is - if any of this "transformative AI revolution" stuff is right at all, then at some point GDP is going to go crazy (even if it's just GDP as measured by AIs, after humans have been wiped out). Paul thinks it will go crazy slowly. Right now world GDP doubles every ~25 years. Paul thinks it will go through an intermediate phase (doubles within 4 years) before it gets to a truly crazy phase (doubles within 1 year).

https://astralcodexten.substack.com/p/the-low-hanging-fruit-argument-models A followup to Contra Hoel On Aristocratic Tutoring: Imagine scientists venturing off in some research direction. At the dawn of history, they don't need to venture very far before discovering a new truth. As time goes on, they need to go further and further. Actually, scratch that, nobody has good intuitions for truth-space. Imagine some foragers who have just set up a new camp. The first day, they forage in the immediate vicinity of the camp, leaving the ground bare. The next day, they go a little further, and so on. There's no point in traveling miles and miles away when there are still tasty roots and grubs nearby. But as time goes on, the radius of denuded ground will get wider and wider. Eventually, the foragers will have to embark on long expeditions with skilled guides just to make it to the nearest productive land. Let's add intelligence to this model. Imagine there are fruit trees scattered around, and especially tall people can pick fruits that shorter people can't reach. If you are the first person ever to be seven feet tall, then even if the usual foraging horizon is very far from camp, you can forage very close to camp, picking the seven-foot-high-up fruits that no previous forager could get. So there are actually many different horizons: a distant horizon for ordinary-height people, a nearer horizon for tallish people, and a horizon so close as to be almost irrelevant for giants. Finally, let's add the human lifespan. At night, the wolves come out and eat anyone who hasn't returned to camp. So the the maximum distance anyone will ever be able to forage is a day's walk from camp (technically half a day, so I guess let's imagine that everyone can teleport back to camp whenever they want).

https://astralcodexten.substack.com/p/idol-words The woman was wearing sunglasses, a visor, a little too much lipstick, and a camera around her neck. "Excuse me," she asked. "Is this the temple with the three omniscient idols? Where one always tells the truth, one always lies, and one answers randomly?" The center idol's eyes glowed red, and it spoke with a voice from everywhere and nowhere, a voice like the whoosh of falling waters or the flash of falling stars. "No!" the great voice boomed. "Oh," said the woman. "Because my Uber driver said - ". She cut herself off. "Well, do you know how to get there?" "It is here!" said the otherworldly voice. "You stand in it now!" "Didn't you just say this wasn't it?"

https://astralcodexten.substack.com/p/who-gets-self-determination I. LSE: Fact-Checking The Kremlin's Version Of Russian History: The notion that Ukraine is not a country in its own right, but a historical part of Russia, appears to be deeply ingrained in the minds of many in the Russian leadership. Already long before the Ukraine crisis, at an April 2008 NATO summit in Bucharest, Vladimir Putin reportedly claimed that "Ukraine is not even a state! What is Ukraine? A part of its territory is [in] Eastern Europe, but a[nother] part, a considerable one, was a gift from us!" In his March 18, 2014 speech marking the annexation of Crimea, Putin declared that Russians and Ukrainians "are one people. Kiev is the mother of Russian cities. Ancient Rus' is our common source and we cannot live without each other." Since then, Putin has repeated similar claims on many occasions. As recently as February 2020, he once again stated in an interview that Ukrainians and Russians "are one and the same people", and he insinuated that Ukrainian national identity had emerged as a product of foreign interference. Similarly, Russia's then-Prime Minister Dmitry Medvedev told a perplexed apparatchik in April 2016 that there has been "no state" in Ukraine, neither before nor after the 2014 crisis. The article is from 2020, but the same discussion is continuing; see eg the New York Times' recent Putin Calls Ukrainian Statehood A Fiction. History Suggests Otherwise. I'm especially grateful to the Russian nationalist / far-right blogosphere for putting the case for Ukraine's non-statehood in terms that I can understand:

https://astralcodexten.substack.com/p/information-markets-decision-markets [thumbnail image credit: excellent nature photographer Eco Suparman, which is a great name for an excellent nature photographer!] Information Markets Niels Bohr supposedly said that "prediction is very difficult, especially about the future". So why not predict the past and present instead? Here's a recent market on Manifold (click image for link). Taylor Hawkins is a famous drummer who died last weekend under unclear circumstances. This market asks if he died of drug-related causes. Presumably someone will do an autopsy or investigation soon, and Chris will resolve the market based on that information. This is a totally standard prediction market, except that it's technically about interpreting past events. Same idea, only more tenuous. We know someone will do an autopsy on Taylor Hawkins soon, and we probably trust it. But how do we figure out whether COVID originated in a lab? This question's hack is to ask whether two public health agencies will claim it. If we trust the public health agencies, we can turn this mysterious past event into a forecasting question.

https://astralcodexten.substack.com/p/highlights-from-the-comments-on-justice A lot of comments on Justice Creep fell into three categories: First, people who thought some variety of: yes, all this stuff is definitely a justice issue, and it's good that language is starting to reflect that more. For example, Adnamanil: So... as someone who actually does use "___" Justice, quite frequently, I'd like to say that I think it's a good thing to reframe "helping the poor" or "saving the poor" as "pursuing economic justice." I don't think it's a good thing for people to think of themselves as saviors, to me that's a really unhealthy and unhelpful mindset which results in people who aren't themselves poor thinking they can be the experts and the decision-makers, and that there is something wrong with poor people, that they need to be "saved" or "fixed." We live in a world where there is enough food to feed everyone, yet people go hungry; enough shelter to keep everyone warm, yet people go cold. To me, that says there is something wrong with our system of resource distribution, not with the people who ended up, for one reason or another, being left out of it. Does that result in a sense of responsibility to fix the system? Yes! Does it imply that we don't live in Utopia? Yes! Because we don't. And I don't think we should pretend to. But it also implies that we *could* live in utopia. It demonstrates a real hope about the possibility of utopia. It says, "if we could figure out how to live together better, we could all have enough to eat and be warm." And Philosophy Bear, as Economic Justice And Climate Justice Are Not Metaphors: Regardless of whether it is useful -and I hope it is- I think that honesty compels a clear-eyed person to talk about many of these things in terms of justice, even in the narrowest conception of justice. The mistake in Scott's article is assuming that these forms of justice are merely metaphors or analogies on criminal justice. Many of these are about justice in exactly the same sense that crimes are about justice- no metaphor required. Of course, they are also about being just in other senses- justice was never just about crime. For example, one can detect demands for social justice in the bible that go far beyond "wouldn't it be nice to help people", but nonetheless aren't framed in terms of the criminal law. Nevertheless, yes, climate justice and economic justice- for example- are also about being just in the same way laws against murder are- no stretching of meaning is required…

https://astralcodexten.substack.com/p/contra-hoel-on-aristocratic-tutoring I. Erik Hoel has an interesting new essay, Why We Stopped Making Einsteins. It argues that an apparent decline in great minds is caused by the replacement of aristocratic tutoring by ordinary education. Hoel worries we're running out of geniuses: Consider how rare true world-historic geniuses are now-a-days, and how different it was in the past. In "Where Have All the Great Books Gone?" Tanner Greer uses Oswald Spengler, the original chronicler of the decline of genius back in 1914, to point out our current genius downturn […] There are a bunch of other analyses (really, laments) of a similar nature I could name, from Nature's "Scientific genius is extinct" to The New Statesman's "The fall of the intellectual" to The Chronicle of Higher Education's "Where have all the geniuses gone?" to Wired's" "The Difficulty of Discovery (Where Have All The Geniuses Gone?)" to philosopher Eric Schwitzgebel's "Where are all the Fodors?" to my own lamentation on the lack of leading fiction writers. If you disagree, I'll certainly admit that finding irrefutable evidence for a decline of genius is difficult—intellectual contributions are extremely hard to quantify, the definition of genius is always up for debate, and any discussion will necessarily elide all sorts of points and counterpoints. But the numbers, at least at first glance, seem to support the anecdotal. Here's a chart from Cold Takes' "Where's Today's Beethoven?" Below, we can see the number of acclaimed scientists (in blue) and artists (in red), divided by the effective population (total human population with the education and access to contribute to these fields).

https://astralcodexten.substack.com/p/mantic-monday-32122 Warcasting Changes in Ukraine prediction markets since my last post March 14: Will Kiev fall to Russian forces by April 2022?: 14% —→ 2% Will at least three of six big cities fall by June 1?: 70% —→ 53% Will World War III happen before 2050?: 21% —→20% Will Russia invade any other country in 2022?: 10% —→7% Will Putin still be president of Russia next February?: 80% —→ 80% Will 50,000 civilians die in any single Ukrainian city?: 12% —→ 10% Will Zelinskyy no longer be President of Ukraine on 4/22?: 20% —→15% If you like getting your news in this format, subscribe to the Metaculus Alert bot for more (and thanks to ACX Grants winner Nikos Bosse for creating it!) Insight Prediction: Still Alive, Somehow Insight Prediction was a collaboration between a Russia-based founder and a group of Ukrainian developers. So, uh, they've had a tough few weeks. But getting better! Their founder recently announced on Discord: I myself am (was?) an American professor in Moscow. I have been allowed to teach my next course which starts in 10 days online, and so I am moving back to the US on Sunday, to Puerto Rico. Some of our development team is stuck in Ukraine. I've offered to move them to Puerto Rico, but it's not clear they'll be able to leave the country anytime soon. Progress with the site may be slow, but obviously that's not the most important thing now. And: I am now out of Russia, and on to Almaty, Kazakhstan. The people here are quite anti-war. I fly to Dubai in a bit. It was surprisingly difficult (and expensive) to book a ticket out of Moscow after all the airspace closures.

https://astralcodexten.substack.com/p/highlights-from-the-comments-on-zulresso Thanks to everyone who commented on Zounds! It's Zulresso and Zuranolone and on the followup Progesterone Megadoses Might Be A Cheap Zulresso Substitute. I'm constantly impressed by the expertise of commenters here and on how much better the biomedical comment threads are compared to some of the others. Among the things I learned: — Metacelsus (who writes the blog De Novo) doubts the price estimates I posted: There's no way it costs $10,000 to $20,000 a gram at scale. Those 3 chemical supply companies specialize in having a very large catalog of small quantities of chemicals for biologists to test in their experiments. (I have personally ordered from 2 out of those 3 for my research.) The price they charge per gram is not competitive at all. He also wrote a longer blog post about the science of progesterone here. — Douglas (who writes the blog A Mindful Monkey) clears up some mechanism details I missed: From Stahl's: 'the precipitous decline in circulating and presumably brain levels of allopregnanolone hypothetically trigger the onset of a major depressive episode in vulnerable women. Rapidly restoring neurosteroid levels over a 60-hour period rapidly reverses the depression, and the 60 hour period seems to provide the time necessary for postpartum patients to accommodate their lower levels'. So the idea is the taper of the steroid is a helpful part.

https://astralcodexten.substack.com/p/justice-creep Freddie deBoer says we're a planet of cops. Maybe that's why justice is eating the world. Helping the poor becomes economic justice. If they're minorities, then it's racial justice, itself a subspecies of social justice. Saving the environment becomes environmental justice, except when it's about climate change in which case it's climate justice. Caring about young people is actually about fighting for intergenerational justice. The very laws of space and time are subject to spatial justice and temporal justice. I can't find clear evidence on Google Trends that use of these terms is increasing - I just feel like I've been hearing them more and more often. Nor can I find a simple story behind why - it's got to have something to do with Rawls, but I can't trace any of these back to specific Rawlsian philosophers. Some of it seems to have something to do with Amartya Sen, who I don't know enough about to have an opinion. But mostly it just seems to be the zeitgeist. This is mostly a semantic shift - instead of saying "we should help the poor", you can say "we should pursue economic justice". But different framings have slightly different implications and connotations, and it's worth examining what connotations all this justice talk has. "We should help the poor" mildly suggests a friendly optimistic picture of progress. We are helpers - good people who are nice to others because that's who we are. And the poor get helped - the world becomes a better place. Sometimes people go further: "We should save the poor" (or the whales, doesn't matter). That makes us saviors, a rather more impressive title than helpers. And at the end of it, people/whales/whatever are saved - we're one step closer to saving the world. Extrapolate the line out far enough, and you can dream of utopia.