The Nonlinear Library LW - Bayesian Injustice by Kevin Dorst
Dec 14, 2023
11:45
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Bayesian Injustice, published by Kevin Dorst on December 14, 2023 on LessWrong.
(Co-written with Bernhard Salow)
TLDR:
Differential legibility is a pervasive, persistent, and individually-rational source of unfair treatment. Either it's a purely-structural injustice, or it's a type of "zetetic injustice" - one requiring changes to our practices of inquiry.
Finally, graduate admissions are done. Exciting. Exhausting. And suspicious.
Yet again, applicants from prestigious, well-known universities - the "Presties", as you call them - were admitted at a much higher rate than others.
But you're convinced that - at least controlling for standardized-test scores and writing samples - prestige is a sham: it's largely money and legacies that determine who gets into prestigious schools; and such schools train their students no better.
Suppose you're right.
Does that settle it? Is the best explanation for the Prestie admissions-advantage that your department has a pure prejudice toward fancy institutions?
No. There's a pervasive, problematic, but individually rational type of bias that is likely at play. Economists call it "statistical discrimination" (or "screening discrimination").
But it's about uncertainty, not statistics. We'll call it Bayesian injustice.
A simplified case
Start with a simple, abstract example. Two buckets, A and B, contain 10 coins each. The coins are weighted: each has either a or a chance of landing heads when tossed. Their weights were determined at random, independently of the bucket - so you expect the two buckets to have the same proportions of each type of coin.
You have to pick one coin to bet will land heads on a future toss.
To make your decision, you're allowed to flip each coin from Bucket A once, and each coin from Bucket B twice. Here are the outcomes:
Which coin are you going to bet on? One of the ones (in blue) that landed heads twice, of course! These are the coins that you should be most confident are weighted toward heads, since it's less likely that two heads in a row was a fluke that that one was.
Although the proportions of coins that are biased toward heads is the same in the two buckets, it's easier to identify a coin from Bucket B that has good chance to land heads. As we might say: the coins from Bucket B are more legible than those from Bucket A, since you have more information about them.
This generalizes. Suppose there are 100 coins in each bucket, you can choose 10 to bet on landing heads, and you are trying to maximize your winnings. Then you'll almost certainly bet on only coins from Bucket B (since almost certainly at least 10 of them will land HH).
End of abstract case.
The admissions case
If you squint, you can see how this reasoning will apply to graduate admissions. Let's spell it out with a simple model.
Suppose 200 people apply to your graduate program. 100 are from prestigious universities - the Presties - and 100 are from normal universities - the Normies.
What your program cares about is some measure of qualifications, qi, that each candidate i has. For simplicity, let's let qi = the objective chance of completing your graduate program.
You don't know what qi is in any given case. It ranges from 0-100%, and the committee is trying to figure out what it is for each applicant. To do so, they read the applications and form rational (Bayesian) estimates for each applicant's chance of success (qi), and then admit the 10 applicants with the highest estimates.
Suppose you know - since prestige is a sham - that the distribution of candidate qualifications is identical between Presties and Normies. For concreteness, say they're both normally distributed with mean 50%:
Each application gives you an unbiased but noisy signal, 𝞱i, about candidate i's qualifications qi.[1]
Summarizing: you know that each Prestie and Normie c...
(Co-written with Bernhard Salow)
TLDR:
Differential legibility is a pervasive, persistent, and individually-rational source of unfair treatment. Either it's a purely-structural injustice, or it's a type of "zetetic injustice" - one requiring changes to our practices of inquiry.
Finally, graduate admissions are done. Exciting. Exhausting. And suspicious.
Yet again, applicants from prestigious, well-known universities - the "Presties", as you call them - were admitted at a much higher rate than others.
But you're convinced that - at least controlling for standardized-test scores and writing samples - prestige is a sham: it's largely money and legacies that determine who gets into prestigious schools; and such schools train their students no better.
Suppose you're right.
Does that settle it? Is the best explanation for the Prestie admissions-advantage that your department has a pure prejudice toward fancy institutions?
No. There's a pervasive, problematic, but individually rational type of bias that is likely at play. Economists call it "statistical discrimination" (or "screening discrimination").
But it's about uncertainty, not statistics. We'll call it Bayesian injustice.
A simplified case
Start with a simple, abstract example. Two buckets, A and B, contain 10 coins each. The coins are weighted: each has either a or a chance of landing heads when tossed. Their weights were determined at random, independently of the bucket - so you expect the two buckets to have the same proportions of each type of coin.
You have to pick one coin to bet will land heads on a future toss.
To make your decision, you're allowed to flip each coin from Bucket A once, and each coin from Bucket B twice. Here are the outcomes:
Which coin are you going to bet on? One of the ones (in blue) that landed heads twice, of course! These are the coins that you should be most confident are weighted toward heads, since it's less likely that two heads in a row was a fluke that that one was.
Although the proportions of coins that are biased toward heads is the same in the two buckets, it's easier to identify a coin from Bucket B that has good chance to land heads. As we might say: the coins from Bucket B are more legible than those from Bucket A, since you have more information about them.
This generalizes. Suppose there are 100 coins in each bucket, you can choose 10 to bet on landing heads, and you are trying to maximize your winnings. Then you'll almost certainly bet on only coins from Bucket B (since almost certainly at least 10 of them will land HH).
End of abstract case.
The admissions case
If you squint, you can see how this reasoning will apply to graduate admissions. Let's spell it out with a simple model.
Suppose 200 people apply to your graduate program. 100 are from prestigious universities - the Presties - and 100 are from normal universities - the Normies.
What your program cares about is some measure of qualifications, qi, that each candidate i has. For simplicity, let's let qi = the objective chance of completing your graduate program.
You don't know what qi is in any given case. It ranges from 0-100%, and the committee is trying to figure out what it is for each applicant. To do so, they read the applications and form rational (Bayesian) estimates for each applicant's chance of success (qi), and then admit the 10 applicants with the highest estimates.
Suppose you know - since prestige is a sham - that the distribution of candidate qualifications is identical between Presties and Normies. For concreteness, say they're both normally distributed with mean 50%:
Each application gives you an unbiased but noisy signal, 𝞱i, about candidate i's qualifications qi.[1]
Summarizing: you know that each Prestie and Normie c...