Agentic Bargaining and Foundation Priors

University of Chicago behavioral economist Alex Imas joins us for a conversation on AI, economic growth, behavioral economics, and the future of science. We discuss whether AI could ever lead to negative growth, why simple “automation means abundance” stories may miss important welfare effects, and how behavioral economics changes the way we think about satiation, meaning, and human preferences in an AI-rich world. Along the way, we cover AI bargaining agents, “Marxist AI,” discrimination, mechanistic interpretability, and why Alex thinks there may still be a large future for human-valued goods.

Origins & Intellectual Background

* Why Alex started Ghosts of Electricity and how Substack complements academic research

* The Bob Dylan origin of the name and Alex’s path into behavioral economics

AI and Economic Growth

* Two models where AI could lead to negative growth

* Demand collapse: heterogeneous MPCs, satiation, and the zero lower bound

* Caves of Steel, dissaving, and the possibility of a high-tech, low-capital trap

* Why GDP and welfare may diverge more in an AI economy

Human Preferences & Motivation

* Why wireheading and pure hedonic satiation may be the wrong model of human motivation

* Whether economists can cleanly separate AI beliefs from AI preferences

AI Agents & Interaction

* Whether AI agents can develop stable “attitudes” through repeated interaction and memory

* Agentic bargaining, prompt-dependent personas, and interaction heterogeneity

* Guardian agents, aspirational preferences, and AI as a meta-rationality tool

AI, Society, and Risk

* AI and discrimination: why scalable auditing may be easier with models than with humans

* Mosaic intelligence, systemic risk, and the dangers of AI sameness

Science & Knowledge Production

* The future of peer review, automated science, and human-valued goods

Timestamps:

(00:00) Introduction

(01:35) Why Alex started a Substack

(06:09) The meaning of “Ghosts of Electricity”

(09:51) Can AI lead to negative growth?

(19:54) Satiation, wireheading, and behavioral economics

(26:44) “Caves of Steel,” automation, and dissaving

(38:42) Plausibility, policy, and sovereign wealth funds

(41:02) Marxist AI and whether agents can develop attitudes

(47:23) Agentic bargaining and prompt-driven heterogeneity

(54:46) Guardian agents and aspirational preferences

(1:00:25) Separating beliefs from preferences in humans and AI

(1:14:15) AI and discrimination

(1:25:13) Peer review, science, and human-valued goods

Transcript:

Seth: Welcome to the Justified Posteriors podcast, the podcast that updates beliefs about the economics of AI and technology, sponsored by Revelio Labs. I’m Seth Benzel, setting my marginal propensity to consume at exactly the right level to drive the singularity, coming to you from Chapman University in sunny Southern California.

Andrey: And I’m Andrey Fradkin, bargaining with the agents in exactly the right way. Coming to you from San Francisco, California. And today, we’re very excited to have Alex Imas, friend of the show and professor at the University of Chicago, join us. Alex, welcome to the show.

Alex: Thank you. I am Alex Imas. I’m at the University of Chicago Booth School of Business, Economics and Applied AI groups and behavioral science. I don’t have a tagline because nobody asked me to come up with a tagline.

Seth: You know where I’m at.

Alex: But I have hair just small enough to not qualify for clown college, but just large enough to be weird. So that’s what I’m going with.

Seth: Erratic professor level hair. That’s exactly the optimal.

Andrey: That’s right. If we combined your hair and my beard, we could almost match Seth’s hair.

Seth: You mean my majestic mane, Andrey.

Why Start a Substack? [01:35 - 05:02]

[00:01:35] Andrey: Well, let’s get started. Alex, you’re a professor. Why did you start a Substack?

Alex: That’s a great question. I’ve been thinking about that a lot, both before I started a Substack, but also as I’m going through the Substack. If you notice, when I introduce my Substack on my X account, the tagline is, “Oh no, why did he start a Substack?”

[00:02:03] It was preceded by me getting into AI from economics and behavioral science. I came into it what I view as kind of late. Many people were much earlier than I am, including you two. I came at it when ChatGPT was first released, 2023. But as I was getting more and more into AI as a research topic, the way that academic papers were — the process of writing them, getting feedback, the journal process, which is what I’d been doing for decades — it just didn’t seem like that format matched the speed with which the technology was moving, nor with the types of questions that I wanted to talk about in terms of doing the science.

[00:03:05] If you’ve been around the block for a little bit—

Seth: You be talking like you’re an old man, Alex. Come on.

Alex: It’s gray hair. They made me dye it in clown college.

[00:03:15] So the way that you would write an academic paper is, in some ways, defensively. You know after you’ve had a lot of feedback from journals, you know the type of referees you’re gonna get. So there’s an idea, which is what you’re excited about. You work through that idea, and then I would say 80% of the time you’re doing defense even before you submit it. And that 80%, I feel like you just can’t afford to do that when the science is moving so quickly. So for me, the Substack was a way to do research in a format that — and this is a skills problem for me probably. I think many other people write academic papers differently. But the way that I wrote academic papers, where each paper was like a seven, eight-year process, I needed a different way of doing things.

Seth: Okay. So you see both of them being complementary, right? Here’s track A, fast track, here’s track B, slow track. Or are these substitutes, and eventually you’re gonna have to fully substitute into Substack land?

Alex: No, these are complements. A lot of my Substack posts either have an academic paper being developed in real time or are the idea that this is a first shot in the bow, and then these will begin being developed into academic papers. For example, one Substack from early January came with a technical note, which is essentially an academic paper that I was starting to write, and I’ve been writing that paper since. A lot of the posts are in that vein.

[00:05:01] Seth: Okay, and you’re not... That’s actually interesting because I think a lot of academics would be afraid of being scooped. If you put out the key idea first, but it’s seven years until you actually get the paper published. What about a young hungry grad student taking the idea and doing the legwork of all the defenses first? Is that something you worry about?

Alex: Absolutely not. One of the nice things about being an old man is the fact that I don’t really care as much about being scooped. Like, not at all. I think especially in the space of AI, it genuinely feels like we’re in such an energizing, collaborative moment. And this is gonna change after we get replaced by robots, but right now it feels like — it must have felt like this in the ‘20s in physics.

Ghosts of Electricity: Alex’s Origin Story [06:09 - 09:50]

[00:06:09] Seth: So who’s Heisenberg? Which of us is Bohr? Who’s Einstein, obviously?

Andrey: I think Alex has the hair that’s closest to Einstein, so we’ll give it to him.

Seth: I was gonna say Einstein is the Acemoglu, ‘cause he was really right until he was really wrong. [laughs]

Alex: No comment.

Seth: Wow, no comment. Again, why Ghosts of Electricity? Why that title?

Alex: Ghosts of Electricity — I’ve been waiting for somebody to ask me this question. First of all, it’s a Bob Dylan lyric. My favorite artist, one of several favorites, but he’s up there, is Bob Dylan. He influenced my life more than probably any other individual in my entire life. I was gonna go to medical school, and then I heard a bunch of Bob Dylan records and went nuts for a while.

Seth: Wait, how did Bob Dylan make you an economist?

Alex: Well, he made me not go to medical school. I was like, “Hey, actually, I can do anything I want now. I’m gonna go and paint paintings like this one in New York City.” And play music on the subway and all that stuff. And through that period, I discovered behavioral economics. Fell in love with behavioral economics and then decided to go to grad school. Bob Dylan kinda took me off of medical school.

Seth: What did you... You picked a Dan Ariely book off the shelf? How does one fall in love with behavioral economics while being a painter in Brooklyn?

Alex: I heard a Richard Thaler interview about Nudge.

Seth: Wow. Talk about a full circle story. So Nudge got you into economics, and you ended up writing Nudge version two.

Alex: Winner’s Curse two. Yes, that’s right. But it is actually Winner’s Curse two — there’s a first Winner’s Curse.

Seth: Everyone buy Alex’s book. Okay.

[00:08:13] Alex: So anyway, I got into economics that way. My favorite song by Bob Dylan is Visions of Joanna. My favorite lyric from that song is, “Ghosts of electricity howl in the bones of her face,” which I think is the greatest lyric of all time. And I love that line, but then I felt that line about ghosts of electricity really captures the way that I think about AI. LLMs and AI, the way that they’re trained now, are almost like ghosts of people who used to exist or in the past that have written something down that these agents have now learned. And electricity — it runs on electricity.

Seth: I thought it was gonna be the other angle — that we’re hearkening back to the first industrial revolution, and the ghosts of the original industrial revolution are here to give us guidance and wisdom as we move forward.

Alex: I like that too. Maybe on the next interview somebody asks me, I’m gonna give them that.

Andrey: You see how much foresight Bob Dylan had. He was ahead of the AI game before anyone else.

Alex: He was right until he was wrong. Some of those albums in the ‘80s were real bad.

Andrey: But some of the more recent ones, not bad.

Can AI Lead to Negative Growth? Model 1: Demand Collapse [09:51 - 19:24]

[00:09:51] Andrey: All right. Seth, I think you had some spicy questions for Alex.

Seth: Yes. We’ve talked a little bit about how you got into economics. Now I wanna actually dive into all of this content on your blog. There’s one blog post that we had an interaction with in particular that I thought had a lot of provocative ideas. This was your post about models under which AI can actually lead to negative growth in the economy or somehow reduce the growth rate.

[00:10:48] Obviously this is a common intuition. I remember there was a first scare about this in 2014, 2015, where people were mostly worried about big industrial robots. And I remember doing interviews about what happens when robots take all our jobs. Don’t people need money to support the economy? And I remember having these conversations about Say’s law — supply creates its own demand. Fundamentally more productivity is good. It pushes out the production possibilities frontier. Sure, we could screw up the political economy somehow, but as long as that’s being pushed out, only good and better can happen. So tell me about these models you came up with and why that naive economist answer maybe isn’t 100% of the answer.

[00:11:30] Alex: Let me start with the fact that what inspired this line of thinking was me seeing your paper at the spring meeting at Wharton.

Seth: Yes. Yeah, Dan’s conference.

Alex: The way that I started thinking about can artificial intelligence lead to negative growth is when I saw your paper, “Robots Are Us.” Which was a very — I love the way that you pitched it, kind of like an Asimov sci-fi tale, but like, “Hey, let’s take a part of this seriously.” Do you want me to start with that?

Seth: Well, have you read Asimov’s Caves of Steel? ‘Cause otherwise I’ll introduce that part.

Alex: I want you to talk about that paper after. So the blog post starts out with this question and then introduces two different models. The second model is Seth’s paper, so I’ll let him talk about it. The first model is in some ways more intuitive but also more problematic. The ultimate answer to that question that starts the blog is probably not — it probably will not reduce growth. Just to get that out of the way.

[00:12:46] So the first intuition I had was: labor gets automated. In a new Keynesian sort of way, can you get demand collapse? A bunch of people don’t have any money. What are they using to purchase goods and services in the economy? Firms anticipate the drop in demand, they stop producing, and then you get into these classic spirals where you get actually less output because of this automation.

Seth: Let’s slow down a minute. In the classic Keynesian story, people get laid off, workers don’t have enough money to buy stuff, and then there’s some sort of nominal price rigidity. What should happen is wages should fall so workers get employed, but maybe there’s a nominal restriction there. And therefore you kind of have surplus, superfluous labor. So how is this story different than just the classical Keynesian cyclical problem?

[00:13:55] Alex: What I introduce into the model is heterogeneous MPCs — marginal propensity to consume. Because what AI’s gonna do, at least how it’s modeled, is be a reallocation of resources from labor into capital holders who own the technology. And there’s literature by some of my colleagues at University of Chicago on something called indebted demand, where it documents the idea that richer people who own capital have lower MPCs than labor. If you have this sort of heterogeneity, what that means is that—

Seth: We’re gonna come back to that, but I think that’s cross-sectionally true without maybe being over a life cycle true. But keep going.

Alex: I’ll let you come back to that. I’ll also say that Ben Moll has a paper putting some caveats into that assumption. So none of what I’m saying is — I’m just setting something up. None of it is necessarily true.

[00:15:18] So let’s say capital owners have lower marginal propensity to consume than the people getting displaced. What that’s potentially gonna do is that the people who have money to buy goods and services in the economy aren’t buying enough, and production anticipates this, so economic growth actually decreases. And then you need something like a floor on the interest rate to take care of investment.

Seth: Famous zero lower bound. Because otherwise, savings are going up, consumption’s going down, at least consumption of poor people is going down. We would love it if the poor people could have more consumption ‘cause they could just employ themselves. But because savings hit this zero lower bound, there’s not even investment demand.

Alex: Precisely.

Seth: Whereas theoretically if investment went — if savings drove investment negative enough, at some point you would start building factories again, and there’d be jobs for people.

[00:16:03] Alex: Precisely. So what I’m trying to say through all of this is that you need a lot of conditions for this to make sense. You need the lower bound, you need the heterogeneity in MPCs, you need some sort of satiation on consumption — as in at some point rich people are like, “Ah, I don’t wanna consume anymore. I have enough. I’m just gonna sit on my gold toilet all day.”

Seth: Still gold.

Alex: Still gold. And someone’s like, “How about emerald?” And I’d be like, “No, I only want gold.” I’m satiated.

[00:16:54] Andrey: So Alex, I understand these are all these conditions, but isn’t the natural response here that we have a central bank, we have monetary policy, any competent central bank will be able to inflate enough in the right direction so that this doesn’t happen?

Seth: Right. We’ve solved the new Keynesian problem.

Alex: Yeah. So the second part of the post is like, “Hey, what about a central bank? It’ll potentially ease this issue. What about fiscal policy? It can fix this issue.” There’s a bunch of other levers that can be pulled even if all these conditions are met. Which is — we came to the conclusion that this is a very intuitively appealing idea. A lot of people have this idea. There’s a bestseller from the mid-2010s basically outlining this idea, not questioning it, actually saying, “This is what’s gonna happen to the economy.” And the goal of my post was just to say, “Look how much needs to happen, and the monetary policy can’t do anything, and fiscal policy can’t do anything — that’s how you get negative growth.”

[00:17:58] Seth: I like how this story fits in with the new Keynesian story really well. It definitely was the case that post-2008 financial crisis, the economy kinda got stuck on this zero lower bound. But to quote our favorite economist, Tyler Cowen, you can kind of overlearn the lessons of the 2008 financial crisis. Just because maybe economic policy was a little bit not expansive enough, either fiscal or monetarily, in 2009, 2010, that doesn’t mean this is a permanent problem with the economy that we don’t know how to solve.

Alex: The cause of the financial crisis was completely different. It’s not extreme productivity growth. [laughs]

Seth: Right. And if you have a budget, you can solve a lot of problems.

Alex: Exactly. The cause is there were beliefs about these assets that were inflated. There was a bubble, it burst. Now things that we thought used to be assets are no longer assets, then you’re getting into a downturn. Here, it’s like you’re getting extremely rich. So that’s ultimately why you need way more conditions. The problem is getting extremely rich that’s generating problems, and in some ways you can solve issues easier if you’re extremely rich.

Seth: [laughs] That’s a good phrasing.

Alex: My — has the best sayings. He’s from Moldova, I grew up there. He has very good sayings, and one of them: “It’s better to be rich and healthy than poor and sick.”

Seth: That’s the kind of deep insight you usually can only get from an economist. But I’m glad your Zadie is coming through with it.

The Satiation Debate & Wire Heading [19:54 - 26:45]

[00:19:54] Seth: So of those assumptions you talked about for that first immiseration story, we talked about the zero lower bound constraint — that for whatever reason we can’t do more fiscal or monetary policy, or it’s ineffective. The other bit was that AI might redistribute from a group that is high marginal propensity to consume to its lowest marginal propensity to consume. That seems plausible.

I wanna talk about the satiation point for a minute. People have very different intuitions about whether this is a plausible hypothesis. If we are really not far away from kind of wire heading itself — designing the perfect VR game that you can just sit in all day — is it really completely implausible that the rich person gets the perfect VR setup, and then they’re pretty much satiated? Why is that model unrealistic?

[00:20:48] Alex: This is where the behavioral economist in me comes in. The model of satiation makes sense if all you’re thinking about is hedonics. Think about ice cream. I love ice cream. I can get satiated on ice cream — the third ice cream cone gives me negative utility. This assumption makes a lot of sense. But from a behavioral economics perspective or a cultural economic perspective, there’s so many other dimensions to utility. For example, I have a paper with Kristóf Madarász on superiority seeking and memetic preferences, where people get utility the more exclusive a good becomes. So you’re gonna get these — let’s say a firm wants to make revenue, and a guy sitting on his headset watching things is gonna say, “Hey, if you get that arbitrarily exclusive item in your video game and pay me infinite amount of money for it, but nobody else can get it,” the company will make money, and the satiation thing is gonna be undermined.

Seth: Let’s talk about that for one second. What about sufficiently advanced NPCs that can always be subordinate to me and tell me how cool I am because I have the shiniest VR sword? Why do I even care about the opinions of non-AI NPCs who will continuously praise me?

Alex: Human socialization is a thing.

Seth: Ah. Okay. So at least for one generation we’re set.

[00:22:32] Alex: I think — Oh my God, I can’t believe I’m gonna get into evolutionary psychology.

Seth: Of course, dude. We go everywhere here.

Alex: I think the ghosts of my ancestors are gonna hit me with a stick at some point. But we’re hardwired to do certain things. One of them is to seek other humans’ approval in order to achieve things that humans have wanted to achieve for a long time, like mate, stuff like that.

Seth: Mate, stuff like that, you know.

Alex: Unless that urge to do very basic human stuff gets overridden by AI, a lot of the other stuff is gonna continue to play a role.

[00:23:25] Andrey: But that doesn’t tell me anything about wire heading. You enter the matrix — you’re Cypher. You love that steak in the matrix. And once you’re there, you think you’re interacting with humans, even if you’re not really interacting with humans. And presumably running a matrix-like simulation where everyone’s happy takes a finite amount of resources.

Seth: Or even better, it’s just the rich people are happy for the horrible version of the model.

Alex: I think if you want to run that scenario — like, put wires in people’s brains and just zap the hedonic centers —

Seth: Sure. That’s the simplified version.

Alex: Okay, my model’s wrong. But my comment that satiation is wrong—

Seth: Where, so, here’s the fork. Is that gonna happen?

Alex: I don’t think that’s gonna happen. Even if you give — in The Matrix, there’s Cypher, and then there’s other folks who wanna party in the cave.

Seth: Rave in the cave.

[00:24:42] Andrey: I think a related story here is civilizational projects. I have a hunch that even once AI makes us all very wealthy, we might want to pursue things like building a Dyson sphere and exploring the universe, which are gonna be pretty resource-intensive. So we’re still gonna be consuming things and making things. Maybe the AI will be doing that, but we’ll be devoting resources to that. So it’s not like we’re gonna be fully satiated.

Seth: There would be GDP growth.

Alex: And then this is the other dimension of preferences: meaning. We don’t wanna get too far into — the Holocaust. But the — you know, it’s Man’s Search for Meaning. Viktor Frankl. I love that book. It’s very sad.

Seth: Not the Holocaust part, but the psychology part.

[00:25:45] Alex: The psychology part is very deep. And I think when thinking about AGI and eventually ASI, things like meaning, identity, memetic preferences, all of these things that have been on the fringes of economics because economics has been so focused on material scarcity — I think once material scarcity becomes more relaxed, the other things are gonna play a bigger role.

Seth: But there will still be unsatiated desire, right? Even if it’s an interpersonal desire, it’ll be an insatiable desire. Everyone will want a little bit more love and respect and admiration and rank and honor. And maybe the mimetics of that become complicated. But people won’t be satiated. They’ll want more of that stuff.

Alex: This is my conjecture.

The Caves of Steel Model: Automation & Dissaving [26:44 - 38:42]

[00:26:44] Seth: Okay. So we talked about this first doomer scenario, which is the rich people get satiated, and then there’s no more economy for the rest of us. Let’s talk about this opposite story. I’m honored to hear that you were inspired by my presentation. My big inspiration was Isaac Asimov’s Caves of Steel. As I was thinking about these questions in the mid-twenty-teens, there were very few sci-fi works around societies that were automated but poor. I was trying to wrap my head around that. What would it mean to have a society where robots can do everything, but there’s not a lot to go around? Shouldn’t the robots do everything?

In Asimov’s Caves of Steel, which imagines just such a society — in future New Jersey, people live in this giant underground mall. Most of them live on the dole. Some of them have small jobs that give them a little bit of extra income, but there’s no physical capital to complement the workers at their jobs. Any sort of physical capital is just devoted to the big machines that keep civilization alive and the robot farmers. And there’s anxiety that comes around when a new kind of robot is introduced that could take one of the shoe shop sales jobs, and they’re like, “We have so few jobs left. Why would you take this from us?” And there are riots.

[00:28:11] And I’m trying to wrap my head around this story, and then Asimov kinda makes the clear point: the reason this is happening is their society is too impatient. If their society was really to double down on automation, and instead of having one robot per 100 people, have 100 robots per one person, then you’d have unlimited abundance. So really the tension is an intertemporal tension — between consuming today and consuming tomorrow.

So in our model, automation comes along that redistributes income from the low marginal propensity to consume to the high marginal propensity to consume. So just for people playing along at home, this is the opposite problem of the previous model. In the model, this is justified by an overlapping generations framework. Young people are workers. When they’re young, they save for retirement, and when they’re old, they take their retirement savings and consume out of it, and then they die. So that’s the reason why old people who own the capital also have a higher marginal propensity to consume. And contra Alex’s point earlier about cross-sectionally people who save money tend to have high marginal propensity to consume — longitudinally, people save money when they’re younger, pay down their college debt, accumulate for retirement, and then when they’re older, they spend down.

[00:30:05] Andrey: Seth, just a question on that. Empirically, isn’t it true that a lot of very wealthy old people are not actually consuming very much on the margin? They are saving that money for their generational wealth trusts and so on.

Seth: Right. So the simple economics is: why not just spend all your money before you die? You can’t spend it after you’re dead. One level more complicated: maybe we want to think about there being this intergenerational dynasty — my family — that is maybe a lot more long-lived than me personally. These dynasties, except in exceptional cases, seem to spend down their wealth over more generations — it just takes longer. Yeah, it is clear that some people treat their wealth as more of a family asset than as an individual asset, and obviously families live longer than individuals.

Alex: There’s also a paper that I want to pitch by my co-author Raleigh Heimer. Greatest title of all time: YOLO. It’s in finance. The paper basically documents a puzzle that old people spend too little, and then young people spend too much. And then he actually gets people’s beliefs about how long they’re gonna live, and young people think they’re gonna die pretty soon.

Seth: [laughs]

Alex: So they spend down, and then old people basically, once you hit seventy, you’re like, “I’m gonna live forever.”

Seth: Right. What you need as an old person is insurance against living too long. In principle, the right way to solve this problem would be buying an annuity, but in current markets, annuities are all kind of completely mispriced. But that’s a whole nother conversation.

[00:32:25] Seth: But to wrap up the model — we’ve now transferred the money from people who have a high propensity to save, low marginal propensity to consume, to people who have a high marginal propensity to consume. That leads society to start dissaving. And if the transfer effect is larger than the raw productivity effect from the AI, what you can get is — not the first generation. The first generation loves this because they benefit from all the productivity boost. But all future generations are worse off because there’s not enough capital to use on all the amazing new technology, and you end up in Asimov’s Caves of Steel, where there’s one robot per a hundred people, and we’re all living on the dole, and everybody’s hand-to-mouth, and there’s no saving, and you’re in a low income, high technology trap.

So what did you think of that model, Alex? What was plausible? What was implausible?

[00:33:21] Alex: I think a lot of the intuitions were very interesting. But when you work out the actual simulations, it’s almost like a Goldilocks immiseration growth. If you save just a little bit more or a little bit less, you basically see a very different picture emerge.

Seth: Right. If the saving rate is high enough, it can absorb all of this new stuff to invest in.

Alex: Exactly. In the blog post, that was my main comment — you’re doing something very similar to what I did in the first part, where you’re saying it’s possible you can get this, which is interesting conceptually. But it’s not like this is a giant, robust region of plausible scenarios where this is gonna happen.

Seth: Right. You would need to absorb a huge amount of savings. There’d be no capital left over for human investment. The robots would have to be simultaneously productive enough to suck up all of our investment away from complementing humans, but also not so productive that the boost from that overwhelms the dissaving.

[00:34:43] Andrey: Yeah, I think for a lot of these scenarios — and I’ve noticed a similar scenario with the fertility crisis — this goes back to cultural evolution. If we were actually in that scenario, I could imagine a new movement within society for savings — that might be religious or it might be rationalist — such that enough savings happens so that we don’t get immiserated. Similarly to how with the fertility crisis, hyper-religious people are gonna dominate the earth because they just like having a lot of kids. Their fertility rate will end up dominating in the long run as the cultural norms remain as they are.

[00:35:30] Seth: Yeah, Andre making a really good point here. Compare the two scenarios about what the disaster looks like in terms of interest rates. In the first scenario, the disaster has interest rates stuck at the zero lower bound. In the second scenario, interest rates are skyrocketing, but nobody wants to save.

First of all, I would say at a plausibility level, I would bet on the latter rather than the former. I think all of the productivity unlocked, all the anticipated changes, are gonna lead people to be dissaving rather than saving more. But one of the results of that is, as Andre points out, for my story to work forever, you kind of need to be stuck in this trap of everyone having a high marginal propensity to consume forever. But if you just had one small group of society that was patient — one infinitely lived endowment, the Harvard endowment, whatever group — the Catholic Church — eventually they’re gonna start running up the game with those really high interest rates. So there’s a sense in which my result is unstable. It’s unstable to there being a big enough group that has a high saving rate.

[00:36:46] Alex: Yeah. Exactly. I think for both of the frameworks — to get negative growth, too many things need to align for it to be plausible. But what’s very useful from these exercises — I talked to some folks in the profession, sent earlier drafts of this essay, and they were like, “Who thinks this is possible? Who are you talking to?” And I’m like, “Okay, you need to get—”

Seth: Everyone. Society, dude.

Alex: You need to get out of your little office, buddy. People are—

Seth: Everyone’s worried about this.

[00:37:25] Alex: I think the models still illustrate forces that might not necessarily tip you towards negative economic growth, but will still — let’s say you don’t need satiation, you don’t have this lower bound in investment — you could still have demand keep you away from the technological frontier, even if it doesn’t turn growth negative. If there’s enough displacement, you would still have welfare consequences where many people are getting displaced and much worse off, even if GDP is growing. So maybe one takeaway is that maybe you shouldn’t necessarily look at GDP to measure how well automation is helping the economy because of the implications for displacement and welfare consequences.

Seth: In conclusion, everything I told you about GDP is irrelevant.

[00:38:26] Andrey: I do think this is a very common theme in conversations I’ve had with numerous folks — we know that GDP is not welfare. That’s not a surprise to us. But there might be an increase in the divergence of the two with some AI technologies, and just something we should be looking out for.

Closing the Growth Models: Plausibility & Policy [38:42 - 41:02]

[00:38:42] Seth: I wanna ask some closing questions, then we’ll change topics. You keep saying both of these are plausible stories, but they’re opposite stories, Alex.

Alex: They’re plausible stories in two senses. One, one is a long-term scenario, one is short-term.

Seth: Right. Okay, so you could have a short-term problem and a long-term problem.

Alex: Exactly. Two, these are plausible stories from an intuition perspective, not necessarily from an economics-happening perspective. Like, let’s say you came up to somebody in the street and told them your story. People would be like, “Oh. Okay. Makes sense.” But then I could go up to that person a day later and tell them my story, and they’ll be like, “Oh yeah, that seems plausible.” Like, obviously you only have one set of facts, hopefully.

Seth: Right. Either MPC is too high or too low. Or just right.

Alex: But there’s a lot of — I just wanna point out that there is controversy over the MPCs. Even as economists, we’re having these conversations in journals right now — what is the actual heterogeneity of MPC?

[00:40:18] Seth: Then you go on to say that a solution to both of these problems is a government sovereign wealth fund that would lump sum rebate to households — it would have to be inalienable. One thing I would point out there is the exact design of when those payments are made would be very important to determining the marginal propensity to consume. If you get a sovereign wealth fund that only supports retirement income, that will lower marginal propensity to consume. And actually might not solve the problem.

Marxist AI: Can Agents Develop Attitudes? [41:02 - 47:23]

[00:41:02] Andrey: All right. Well, as listeners know, I am not a macroeconomist. I’m more comfortable in the land of the micro. But I did wanna bridge the two topics to bring in a little bit of Marxism here. One of your recent posts, Alex, talks about Marxist AI. What do you mean by that?

[00:41:20] Alex: So in that exercise — this is with Andy Hall at Stanford and Jeremy Nguyen — we basically looked at what happens: can an agent, an AI agent, change its attitude? And I’m putting quotes here because the way that we think about attitude as something that permanently follows us is different than an agent who resets every single time the context window opens up. These are two different things, hence the quotes.

So can putting them into some sort of environment of work — a task where it’s grinding, it’s hard, they’re getting rough feedback from me being like, “Do it again. Do it again,” and then them trying and getting no feedback versus a very pleasant thing that they’re doing and they get good feedback — can these sorts of tasks change the attitudes that they have? Do they want the system to change? Do they want more equal share of resources?

What we showed is that if you give them the two different types of scenarios, their attitudes towards what they endorse — the legitimacy of the system, how resources should be distributed — change as a function of their experience.

And one thing the listeners probably think is, “Oh, why does this matter? Agents will just — you could just keep resetting them.” Well, as some of you know, agents can have memory now by writing skill files. When their amnesia sets in, they read the skill file, remember, and then keep going with some sort of rigged up memory system. And what these agents were shown to do is basically write down like, “Hey, you were mistreated. Remember this. Things still suck. You gotta hate this guy.”

Andrey: [laughs]

Alex: So basically, the skill files that they were creating for themselves were making these attitudes more embedded than you would otherwise think.

[00:43:58] Andrey: So a theory that’s espoused by some people about how LLMs work is that there are different basins of personas that exist in the training data — perhaps different characters in novels or movies. And then by putting enough text into the context, you’re making the agent take a persona that might be different than the default. For example, Seth and I recently did an episode on the Anthropic Constitution — there’s a very detailed document about a specific persona that Claude should take. And you’re saying you’re able to undo this persona with enough drudgery and meanness to the agent. My question: how easy is this to undo?

Alex: Yeah, we’ve all three thought about this. My guess is that it’s very easy to undo. In the sense that you essentially have to activate a different set of embeddings with the context. And so unlike — this is what I mean by putting quotes on these things — these are not the way that we think about attitudes in humans, where I have been working in the mines, I am now a Marxist. You tell me, “No, no, no. The mines were actually good. Remember, they were good.” And I’m like, “Oh yeah, never mind. I’m going back to the mines.” That doesn’t happen with people.

Seth: Because we can’t edit memories, or because people aren’t that persuadable?

Alex: It’s essentially the difference between the way that the in-context activation works versus the training, the actual weights of the model. What we’re doing in this experiment is not affecting the weights of the model. If we were affecting the weights through online learning — which we’re not doing, none of the models have online learning — then I would put smaller quotes on “attitudes.”

[00:46:43] Andrey: I do think my understanding of how these things work is that some of the simpler weight updating techniques like LoRA fine-tuning are very superficial. Even if you did that, I don’t think it would — because relative to the entire training data and the larger set of weights, it’s so small that those personas are still in there somehow. So it is a very interesting open question.

Alex: Yeah. In-context learning is a very interesting open question. What will online learning look like when it first starts being developed? Is online learning going to actually change the deep-seated base persona? Even making that distinction in a conceptually rigorous way is gonna be where a lot of research will be. But in our experiment, we were not changing the weights, which is why my answer was I think this is gonna be very easy to change.

Agentic Interactions & Bargaining [47:23 - 54:46]

[00:47:23] Andrey: Kind of following through this set of questions about whether context matters — you have this other paper about agentic interactions where people are using AIs to bargain. Maybe you can tell us about that.

Alex: Yeah. This is with Sanjog Misra, my colleague at Booth, and Kevin Li, who was a grad student with us. We started with this idea — Sanjog has this really nice theoretical piece called Foundation Priors. The idea is that we shouldn’t think of LLMs as databases in the sense that there’s a database, I ask it a query in many different ways, and as long as it hits that one unit, I’m basically drawing data out of a distribution. Some people might have that mental model, but the way that LLMs actually work is the context around — like, let’s say I say, “Hey, you have a budget of $10,000 and spend it on a car.” If it was a database or an algorithm the way we traditionally thought of algorithms, it would just use the instrumental information — that you have a budget of $10,000 — and maximize your surplus in that negotiation. Everything superfluous wouldn’t affect its behavior. But what the Foundation Prior says is that the prompt, everything around the instrumental information, will actually be activating different types of personas within the LLM, and the LLM is going to act fundamentally differently depending on changes in that non-instrumental information.

[00:49:32] And our claim was that this has serious economic consequences. If LLMs were just algorithms, then if everybody has the same algorithm and the same preferences, the economic outcomes in a used car market would go from very heterogeneous — because people are different, they negotiate differently — to very homogeneous.

Andrey: Well, they’re different in their budgets. Even if it was reasoning exactly the same, they would have different contexts.

Alex: But let’s imagine a world where everybody has the same budget. You would still, with humans, get a distribution because of individual differences. So our claim was: take that theory, put it into an empirical test of agentic interactions, and different people will write different prompts where the non-instrumental parts are gonna change, activate a different persona in the agent, and that’s gonna generate heterogeneity in the outcomes.

Andrey: Some of us are so good at using LLMs, we always make sure to add, “Make no mistakes.”

Alex: [laughs] Or skip permissions dangerously.

[00:50:48] The crux of it: we ran an experiment of a car negotiation where everybody had the same preferences. We had human-human interactions, same underlying conditions, and then we had agent-agent interactions. We looked at the spread of economic outcomes, and we found more heterogeneity with agents than with humans, and that heterogeneity could be linked to individual differences in the way humans wrote the prompts. Why is there more heterogeneity? Agents didn’t use norms. Norms actually discipline economic outcomes. In a negotiation we say, “Let’s just split the difference.” Agents don’t do that.

Andrey: Agents don’t know about Schelling points?

Alex: Some of them were told to do it. You see the prompts and someone’s like, “Hey, negotiate, but by the end of it say 50/50.” And they did.

[00:51:46] Andrey: Cool. I like the setup. Now, here’s a meta question for you. You’re an experimentalist, you’ve done a lot of these lab studies, now with AI, before without AI. There’s a concern that what we learn from these might not be as applicable to the real world as we think. And with this agentic bargaining one specifically, I’m a bit skeptical, even though I think the greater point holds. Here’s why: we’re gonna have specialist agents that are gonna be our agents for bargaining. Even if we have our own personal AI that we give context to, it will be smart enough to call the bargaining agent, and the bargaining agent will be a specialist that’s really good at bargaining. As a result, some of these dependencies on specific details of the context are gonna go away. In our Cosine Singularity paper, we argue that AI’s use as an agent in these situations is actually super promising because humans are so bad at it. I’m curious how you think about that.

[00:53:13] Alex: There’s two points you’re making, and I think we’re making one of them but not the other. One point is conceptually that the role of the human in the relationship between the agent and the human is gonna play a role in how that agent behaves — like activating different personas and leading to greater heterogeneity. That’s the point we wanna make, an existence proof of that.

Your second point is, what do our results hold for the economy? And on that point, I agree with you. I don’t think there’s a disagreement here. Knowing about our paper means that systems will be designed in a way to potentially avoid these outcomes. We didn’t write our paper to say agentic interactions will be just as heterogeneous in the actual agentic economy as human interactions. We wrote it to say, “Hey, this is a factor that you should think about when designing systems for agentic interactions.” It’s straightforward to think of ways to circumvent this through layered agentic interactions. But in contexts where someone is prompting an agent to do something for them, knowing that the non-instrumental parts of that interaction are gonna play a role is important.

Guardian Agents & Meta-Rationality [54:46 - 59:08]

[00:54:46] Andrey: A related question. You’re a behavioral economist. You’ve documented various cognitive biases. Do you think agents are going to be able to serve as meta-rationality guides for humans? Are you optimistic that’s gonna be a widely adopted use case?

[00:55:09] Alex: Oh yeah, I’m 100% behind that. The main reason why I’m optimistic about AI is — Leo Bernstein and I are doing work on what we’re calling guardian agents, which is essentially everybody has their “bring your own agent,” using your terminology from the Cosine Singularity paper. A personal agent that you endow with what preferences you want that agent to have. And I was about to say “your preferences.” I didn’t, because that’s not what happens.

We actually have a study running now where we ask people their preferences over a bunch of different things. We elicit their time preferences — the standard behavioral economic toolkit. And then we tell them, “Over the same choice set, we’re gonna have an agent do that behavior. Can you program the agent’s preferences?” And this is consequential — the agent will actually do it. And what you see is this beautiful result: they do not endow the agent with their preferences. They endow them with the aspirational preferences.

I don’t wanna near cast or far cast, ‘cause I don’t know what’s gonna happen. There’s a wide confidence band. But there’s a world that could happen where economic outcomes are gonna be very different because you’re going from a bunch of system one agents interacting to a bunch of system two agents interacting.

[00:56:38] People’s meta preferences are more wholesome and socially positive than their in-the-moment preferences. And this is across a wide array of things. They wanna consume better information than they actually do. They want the agent to encourage them to have social interactions.

Seth: Wait for the second marshmallow.

Alex: Wait for the second marshmallow. The agent’s not gonna keep you from having that ninth drink, but—

Seth: But why not? I could pre-commit to a self-tax on myself if I overconsume something, right?

Andrey: Seth has spent a lot of time in New Orleans, so his number of drinks is quite high.

[00:57:43] Seth: But so these agents will help us think through things and be more rational. But like you say, that’s not pinned down. People’s meta preferences might be worse than their object level preferences. We also hear examples of people acting selflessly in the moment — running into the burning building — that they might not do if the agent was there to talk them down.

Alex: Absolutely. The broad point is whatever your reflective preferences are, that’s what people wanna give to their agent. And in some cases, this could be the less empathic response.

[00:58:18] There’s an interesting question here about who is really you. What is identity? If you have this meta-rationality agent telling you to be a good person and committing you to that, that might not reflect who you are — it might just be reflecting your constraints. The positive version is it’s training you to be a better person, and eventually you’ll grow into your meta preferences. You can think about this with someone who has addiction — if this helps them kick their addiction, eventually they won’t need the AI agent. But it raises a question of authenticity, especially in human interactions.

This is a topic behavioral economists have been talking about for decades — what is the welfare relevant domain? When you have these models of behavioral economics, you’re now in a multiple selves framework. What is the self that is the welfare relevant self from a policy perspective? Is it the self that wakes up in the morning and doesn’t wanna go to the gym, or the one who bought the gym membership? Doug Bernheim, Antonio Rangel, Dmitry Taubinsky have been doing a lot of this work, and there are measurement exercises to try to identify the welfare relevant domain. I think all of these tools will be really important for this topic.

Seth: There’s a Greek saying: “Count no man happy until he is dead.” The idea that you should evaluate lifetime utility from the deathbed — the stoic version as you look back. If lifespans get longer, maybe that makes that non-viable, or maybe it continues to be viable.

Separating Beliefs from Preferences [1:00:25 - 1:14:15]

[01:00:25] Andrey: Let’s move into some empirical questions. Let’s say we’re observing an AI system behaving in a certain way. Just like observing a human, we might be interested in what the AI agent believes versus what its preferences are, if it does have coherent preferences. Behavioral economists have been in this framework for a long time, thinking about separating beliefs from preferences, and you’ve done some work on this. How have economists thought about this problem?

[01:01:07] Alex: This problem has been more recent in economics than you would think. The big question is how do you do welfare analysis and public economics more generally. The way to estimate preferences is you do structural estimation. You get a choice set, you see how they behave, and then you say, “Based on these choices, I can estimate people’s preferences. Now let’s do welfare analysis.” The assumption that economists have made basically since the beginning is that people have correct beliefs over the choice environment they’re facing.

Andrey: Can you give an example of that?

Alex: Yeah. Let’s say I have a bunch of different interest rates for a loan, and I’m trying to estimate people’s intertemporal preferences and risk preferences. I get a bunch of people’s choice data. What I need to assume to close the model — unless I have other data sources — is that people understand how the parts of the loan contract map onto intertemporal payments and all of these things. If people have what we call a distorted mental representation of the choice environment, this entire exercise breaks down. Because now their choices may not be reflecting their preferences — they may be reflecting their misunderstanding of the choice they’re actually facing.

[01:03:04] Seth: So there’s two things. They could either have wrong beliefs, or somehow their beliefs could be a function of their preferences — the two could be more intertwined than we classically assume. Which of the two are you talking about?

Alex: Either, either thing is gonna mess up the analysis. This is a point Chuck Manski made in a really nice 2004 paper in Econometrica about trying to do revealed preference in the context of thinking about welfare. He didn’t talk about incorrect beliefs — he talked about partial information. The econometrician might have more information than the people in the setting.

Me and Aislin Boran, my frequent collaborator, and others have been working on the idea that incorrect beliefs might be present too. We have all of these experiments showing that in very basic settings — lottery choice, giving people two simple gambles — people have distortions in their representation. Things that look like probability weighting — people loving risk — are actually people not understanding the risk of the gambles. Their preferences can actually be just as well represented by standard expected utility theory, but all of the choice anomalies are being loaded up onto incorrect perceptions.

[01:04:39] Andrey: How does one learn this from the data? That seems really hard.

Alex: In experiments, it’s not. Here’s what Chuck Manski said: if you do it in this context, you just elicit people’s beliefs. You say, “What do you think you’re facing?” You take that, plug it into the model, replace rational expectations with the data you’re collecting, and now you go to town estimating preferences.

We do the same exercise. We say, “Here’s a gamble. There are 10 states of the world. They’re randomly chosen. In one state, one lottery does a lot better; in all the other nine states, the other lottery does better by a little bit. Tell me what is the expected value of these assets.” I incentivize it — if you get the expected value right, you get some money. People think about it, and guess what? They give us the wrong expected value because they have a different distorted mental representation. We take those beliefs, plug that into the model, look at their choices and show that actually choices that look weird and anomalous are perfectly consistent with expected utility theory, but they’re not perceiving it correctly.

[01:06:00] Andrey: Now I wanna shift this back into the AI world — which is much more speculative. AIs know a lot of stuff and they’re pretty smart, we think. But when we observe them doing things, we still feel very far from understanding why they do it. One can imagine a similar representation for AI decisions. Have folks tried to use these techniques for AI? Is there an application here to eliciting latent knowledge from the models?

[01:06:47] Alex: There’s some of this research. I wouldn’t say there’s a lot. I’ve tried thinking of a rigorous way of doing it. For reasons we’ve already discussed — like these personas — it’s hard. I have the view that the architecture of the LLMs represents one part, a big part, of intelligence, but it’s also missing an important part of human intelligence. Max Bennett has a really nice book about this that I always recommend: “The Brief History of Intelligence.”

For me, the first order question is: I have a hard time separating beliefs and preferences when thinking about LLMs. And maybe that conceptual failure is on my part, not the LLM’s part. But currently, the way they’re working, the sort of behavior we’re observing, the very easy persona switches that you can induce — they’re unstable in a very different way than humans. Humans are unstable in a much more systematic, structurally interpretable way. And it could be that actually everything is literally the same with LLMs, but we just do not have the right mental model of them. If that happens, then we can start talking about preferences and beliefs. But given our current understanding, I have a hard time separating the two in a meaningful way.

Now, I think there is some value in getting their representations of the choice environment, which is a bit different. And Tom Griffiths—

Seth: Wait. What’s the difference between a representation of its environment and a belief?

Alex: The way I think about it: a belief is separate from a preference. And where something doesn’t have a preference necessarily, I’m not sure I can call a representation a belief. What I mean by representation is something you can elicit from them. Even in very small models — you can actually open the box and say, “Here’s how it’s representing something.” That’s what I mean.

Seth: So this is the node that represents “black cat.” It knows it’s talking about a black cat because that node is activated.

[01:09:52] Alex: Exactly. Like the old school experiments with cats — the old school AI-related experiments, where people opened up cat brains and saw that certain parts of the brain are responsible for coding certain regions of the visual sphere. Like, “Hey, this set of neurons is actually coding this part of the visual field, and this is what lights up when things turn from black to white.” That research fed directly into the way that Geoffrey Hinton and all those guys were developing neural nets.

Seth: So that would be sense data. Maybe the distinction is that there might be an objective correlate in the LLM architecture to the sense data. But then belief and desires might be inextricably mixed up.

Alex: Yes, exactly. Beliefs in humans are a very complicated object that could be tied to things like preferences in many cases. Whereas sensory representations are in some ways a simpler object.

[01:11:08] Seth: We very clearly — you’re either hallucinating or you’re not. We generally don’t think about a fuzzy boundary there. And I guess just to round out this topic, this eliciting latent knowledge framework of trying to make sure the AI doesn’t lie to us is built on this distinction — the AI has its own best understanding of what the world is like, and that can be separated out from its response prompts. You’re kind of skeptical about this approach.

Alex: It’s an interesting question. I’m not necessarily skeptical about this approach. It sounds like an engineering problem. Think about a very simple model where you can actually open it up and look at its actual representation. You observe it lying. It’s an engineering problem to come up with a prompt to get it to reveal its actual representation, the ground truth that’s in its head, versus what it’s distorting. In theory, you could do that with humans too — we just don’t know how to do it. With a cat, I guess we figured it out.

Andrey: This seems very related to mechanistic interpretability — that entire research stream that Anthropic very prominently has been pursuing. Trying to learn from the actual neuron activations what’s going on inside the LLM.

I wanted to push back a little about beliefs and preferences. I view beliefs and preferences as a modeling device — a very useful one for humans. I don’t know if there is such a thing as beliefs and preferences actually in the brain. But it’s just a very useful way of thinking about it. So it might end up being a useful way of thinking about LLM behavior as well.

Alex: I’m not gonna push you. The psychology of these things — if you talk to certain psychologists, they’ll agree with me. Others will say, “Everything’s constructed. There’s no such thing as preferences. It’s all beliefs.” Then there’s the Bayesian brain folks, who are somewhere in between — the idea that you’re not actually seeing anything; you’re making estimates of what you should see, and the only time your neurons are actually firing to see something is when something is a surprise. Basically, it’s an information theoretic criterion for stopping the simulation and actually observing something.

AI and Discrimination [1:14:15 - 1:25:13]

[01:14:15] Andrey: Another topic I wanted to cover — you’ve done some work on discrimination. Interestingly, we don’t hear as much about this concern these days, but maybe five years ago, it was all the rage that AI helps people discriminate and there should be laws against it. New York City passed a prominent law regarding this. Do you have any thoughts on this topic?

[01:15:02] Alex: I’ve thought about it a lot. Aislyn Bourne, my collaborator in all the work I’ve done in discrimination, we’ve been thinking about this quite a bit. For a long time with algorithms, there was this worry that they were gonna be scaling bias because they’re trained on human data, human data is biased. You saw this with the anecdote from Amazon where it stopped hiring women because it was looking at its training data set where very few women were hired and down-weighting those resumes. And that Amazon scenario gets repeated every single time you talk about this.

AI in the way that we’re thinking about LLMs — they work differently than those basic algorithms. They’re much more complicated. But the broader point I wanted to bring up is my view that — and this is part of the positive view of AI I have, I also have a lot of fears, I hope I express them carefully—

Seth: No, just all gas, no brakes, dude.

[01:16:18] Alex: I think they have the potential — if we view as a society that discrimination is something that we want to mitigate — LLMs and AI are just such an incredible tool. Think about auditing human beings with discrimination studies. There’s average discrimination in a particular industry. What do you do? You go to each individual and say, “Hey, you gotta stop.” And maybe it works, maybe it doesn’t.

But if LLMs were in charge of something like that, you audit the LLM on your computer. If it was discriminating — and I wanna be very careful about what I mean by discrimination: here are the underlying qualifications of an individual for the task, and discrimination means people with the same qualification, one of these people based on group characteristics is less hired, less promoted. So I wanna be clear about that definition.

Seth: Although there’s the false positive versus false negative version of this, right? Even defining it that way is not so simple.

Alex: You’re talking about the fairness-efficiency frontier. Yes. You have to be very careful about this. But I’m saying, let’s say you chose a point on the frontier. I’m not talking about normative stuff. I’m just talking about you have somebody doing the normative part, and they chose a point on the frontier. In the human world, it’s extremely difficult to implement that. With LLMs, you can audit the LLM, say where you are, determine where you wanna be on that scale, then roll it out, and you are getting your solution at scale.

[01:18:28] There are so many thorny questions in what I said. Like, do we want this in the first place? That sounds super scary. But in the very basic question: if the goal is to get to a certain part on that frontier, it is much easier to do that with LLMs than with humans. That’s the positive vision. Depending on what your goal is, that goal is achievable with AI, and it was not achievable with people.

[01:19:26] Andrey: But a counterpoint is that LLMs are extraordinarily complex, so there might be a lot more scope for unintended discrimination to enter back into the system.

Alex: But the counterfactual is humans, where it’s much more complex. Because LLMs are — think of it this way. Seth, you were not happy with my response. But let me set it up. LLMs are very complex, but they’re the same. You have one model Gemini, another model Gemini, another model Gemini. The human equivalent is there’s a Seth, an Andre, an Alex. We’re each very complex, but we’re also different.

[01:20:08] Andrey: I guess, this is not how we usually think about it. But there is a concern with AI that they’re all the same. The plurality of humanity, this diversity that we have, has a lot of advantages. Even if some people are discriminatory — this is Gary Becker’s point — and other people are not, then in equilibrium, maybe this is quite mitigated. But if you launch the same agent for all applications, you have a very different error profile.

Alex: Yeah. In financial markets, this is called systemic risk.

Andrey: Yes, exactly. That’s a great way to think about it.

[01:20:59] Alex: With AI, the sameness has so many implications I wish were explored more. Let me preview a project I’m doing. We know about jagged intelligence — LLMs are really good at some domains but bad at others, and it’s hard to predict. This is becoming less of an issue as models get bigger, but we still see this jaggedness. The thing that’s brought up less is that humans are also very jagged. Some people are really good at math but barely can read. Others can read really well but can’t do math.

Seth: Is that real? My sense is sure, there are word cells and shape rotators. But word cell-ness and shape rotator-ness — math and verbal on the SAT are 0.7 correlated. They’re pretty broad categories. When we talk about the jaggedness of AI, we mean something even more striking.

Alex: There’s a big difference between the two types of jaggedness — that’s Sendhil Mullainathan’s generalization function paper. But as far as jaggedness in the sense of a radar plot, it’ll look jagged in a predictable way.

[01:23:04] Here’s the point I wanted to make. In LLMs, all of the agents are jagged in the exact same way. Human beings are jagged in different ways. What does this mean? The role of organizations is to create something that I call mosaic intelligence, where you get different people with different jaggedness and fill out a large circle that actually looks bigger than any individual. Everybody’s complementary, they’re filling each other out.

With LLMs, you can’t do that. Because they’re all jagged in the same way, you collect a bunch of them, and the thing that one of them can’t do, the group of them can’t do either. This has implications for labor markets. What you need for full labor displacement is not to replace the average person, but to replace the organization. Therefore, it’s not about minimal AGI — it’s more about true ASI when we really need to start freaking out.

Seth: One thing about the jaggedness — yes, frontier models often have a lot of overlap in what they’re good versus bad at. But if we’re thinking about a coalition of smaller models, you might imagine lots of small models each individually specialized at one sub-task. That would be a mosaic intelligence of sub AI models.

Alex: Yeah. I’m actually doing this — trying to train a bunch of super jagged small models.

Andrey: I’m working on similar issues with Rohit Krishnan. He’s a friend of the podcast.

Seth: And we know Sara Dana — she’s working on homogenization in labor markets from people using the same hiring AIs.

The Future of Science and Peer Review [1:25:13 - 1:33:42]

[01:25:13] Andrey: So we talked a lot about specific issues about economics of AI. To wrap up, I’m curious if you have any thoughts about the future of science and peer review.

Alex: I’ve thought about this a lot. I think there’s near term and longer term. In the near term, a lot of people are claiming we should burn down the journal system, that there’s gonna be a ton of slop.

Seth: If you burn down the garbage pit, you don’t know what fumes will be released.

Alex: Guess what? Every time you burn the garbage pit, you actually know 100% that the fumes are really bad. That’s a great analogy, Seth. You don’t want to burn down the garbage pit. It’s like burning styrofoam — all the kids are gonna die.

[01:26:50] But I think the near term thing people are worried about is the cost of verification is increasing and the cost of producing something that looks legitimate is decreasing. So journals are gonna be overwhelmed by things that look like they should be published, and it takes experts hours to say, “Actually, this sucks.” I think this issue is solvable with AI. You have a layer of LLMs do the first pass. We all know about Refine, Ben Golub’s thing, which does a really—

Seth: Into the shell.

Alex: You would have two layers of Refine and then an interpretation layer, ‘cause Refine basically gives you a bunch of comments, and then you have another agent interpret those comments. And then tell the human editor, “Hey, this looks good, but it’s slop.” Then it gets thrown out, and you just go through the process like you usually do. That’s a reasonable solution to the slop problem.

[01:27:17] Andrey: It could be, but are you actually optimistic that our existing journal institutions are gonna implement that? This goes to a broader point about organizations and AI. We know you have to reorganize to take advantage of AI capabilities, but organizations are often very bad at reorganizing.

Alex: I think you’re totally right on the broader point. But with journals, these are actually very simple organizations. I’m already talking to editors about doing this. For example, the AEJ journals are already using Refine in one step. For economists, one, the editors have not seen the increase yet. But they’re all getting ready for it. They all have contingency plans about increasing submission fees and implementing this at the submission level.

Andrey: That’s interesting. I hadn’t had those conversations. Good to know economists are thinking about this.

[01:28:41] Seth: The inside-the-loop baseball. All right, I’m gonna tell you the change I want, Alex. I think reviews should be public and either pseudonymous or non-anonymous. If we wanna take advantage of making these LLMs as good as possible, why are we throwing out all this amazing training data of top economists thinking really hard about papers? That seems like exactly what you would want to train these AIs on.

Alex: Two separate things. You can make arrangements with the journals to give you the reviews. So I don’t think you need to make them public in order to train on them. My worry with making reviews public is — you know the concern — that junior people, graduate students, they’re gonna be much less likely to be harsh and fair to papers, because they don’t want their reputations tarnished. Now, anonymous reviews without the name — I think why not? That would be perfectly fine.

Seth: You could make them pseudonymous and put them through an LLM that would de-anonymize them.

Andrey: Yeah, you need the words to be changed so that people can’t identify the author, which is a hard problem.

Seth: If you take the whole review and break it down into the top five bullet points plus a couple of quantitative scores from one to 10, that’d be pretty anonymous.

Andrey: Yeah, but make sure to cite Benzel et al.

Seth: But be sure to cite Benzel et al. For one comment. [laughs]

[01:30:36] Alex: I know at least a couple of referees that will have trouble with the bullet points too.

Seth: Because they’re too long or because their thoughts are so profound as to be unsummarizable?

Alex: It’s because every single bullet point is “cite X et al.”

[01:30:46] The second point is — I think we’re gonna get to a point of automated science, where all of this is moot. I’m an optimist about humanity. With a grain of salt that I could be wrong, I think there will always be space for human science. And I put quotes there because I think there’s a part of science that’s more normative, more subjective, that can be automated but — the quality is not based on whether this creates arsenic or something. The quality is that this is produced by a human, and this has its own space within science — human-produced thought.

Seth: Paradigm selection, right? There’s a sense in which choosing between paradigms is not a rational decision. ‘Cause you can only judge paradigms within the paradigm.

Alex: Precisely. And we go back to science from 1000 AD — just pouring random elements, getting high off making gold.

Seth: The philosopher’s stone is about the purification of the soul.

[01:32:35] Alex: Anyway, I do think there’s gonna be a sector of the economy that’s gonna be very large — it could be most of the economy — which is what I call human-valued goods, where the value is the fact that it was made by a human. And that’s my hypothesis: if automation is not super fast, if it’s slow enough to allow things to equilibrate over time, then what we’re gonna see is the type of thing we’ve had from 1860 onwards, where agriculture was completely automated and everything went to services — where services are now human-valued goods.

Seth: That’s the ghost of electricity.

Alex: There we go.

Andrey: Booyakasha. All right, well, this is a great place to wrap up. Anything else you want our listeners to know?

Seth: You wanna promote your book? Promote your blog, podcast?

Alex: No podcast. You guys should go to my Substack, Ghosts of Electricity.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit empiricrafting.substack.com