Calculations and Catastrophe in the Pursuit of Reward

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: All About Concave and Convex Agents, published by mako yass on March 25, 2024 on LessWrong.
An entry-level characterization of some types of guy in decision theory, and in real life, interspersed with short stories about them
A concave function bends down. A convex function bends up. A linear function does neither.
A utility function is just a function that says how good different outcomes are. They describe an agent's preferences. Different agents have different utility functions.
Usually, a utility function assigns scores to outcomes or histories, but in article we'll define a sort of utility function that takes the quantity of resources that the agent has control over, and the utility function says how good an outcome the agent could attain using that quantity of resources.
In that sense, a concave agent values resources less the more that it has, eventually barely wanting more resources at all, while a convex agent wants more resources the more it has. But that's a rough and incomplete understanding, and I'm not sure this turns out to be a meaningful claim without talking about expected values, so let's continue.
Humans generally have mostly concave utility functions in this sense. Money is more important to someone who has less of it.
Concavity manifests as a reduced appetite for variance in payouts, which is to say, concavity is risk-aversion. This is not just a fact about concave and convex agents, it's a definition of the distinction between them:
Humans' concavity is probably the reason we have a fondness for policies that support more even distributions of wealth. If humans instead had convex utility functions, we would prefer policies that actively encourage the concentration of wealth for its own sake. We would play strange, grim games where we gather together, put all of our money into a pot, and select a random person among ourselves who shall alone receive all of everyone's money.
Oh, we do something like that sometimes, it's called a lottery, but from what I can gather, we spend ten times more on welfare (redistribution) than we do on lottery tickets (concentration). But, huh, only ten times as much?![1] And you could go on to argue that Society is lottery-shaped in general, but I think that's an incidental result of wealth inevitably being applicable to getting more wealth, rather than a thing we're doing deliberately.
I'm probably not a strong enough anthropologist to settle this question of which decision theoretic type of guy humans are today. I think the human utility function is probably convex at first, concave for a while, then linear at the extremes as the immediate surroundings are optimized, at which point, altruism (our preferences about the things outside of our own sphere of experience) becomes the dominant term?
Or maybe different humans have radically different kinds of preferences, and we cover it up, because to share a world with others efficiently we must strive towards a harmonious shared plan, and that tends to produce social pressures to agree with the plan as it currently stands, pressures to hide the extent to which we still disagree to retain the trust and favor of the plan's chief executors.
Despite how crucial the re-forging of shared plans is as a skill, it's a skill that very few of us get to train in, so we generally aren't self-aware about that kind of preference falsification towards the imagined mean and sometimes we lose sight of our differences completely.
Regardless. On the forging of shared plans, it is noticeably easier to forge shared plans with concave agents. They're more amenable to stable conditions (low variance), and they mind less having to share. This post grew out of another post about a simple bargaining commitment that would make concave misaligned AGIs a little less dangerous.
In contrast, let's start...