The Nonlinear Library EA - A simple and generic framework for impact estimation and attribution by Christoph Hartmann
Feb 4, 2024
34:56
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A simple and generic framework for impact estimation and attribution, published by Christoph Hartmann on February 4, 2024 on The Effective Altruism Forum.
TL;DR
Building on the logarithmic relationship between income and life satisfaction, we model the purchase of goods as an exchange of money for life satisfaction.
Under that lens, a company's impact on the life satisfaction of its customers is the income-weighted sum of the company's revenue.
By comparing the company's revenue with its cost structure, we can attribute the relative share of impact to the company's employees, suppliers, and shareholders.
While this approach primarily focuses on for-profit companies, it can be extended to NGOs by reframing them as companies selling attribution of their primary impact for donations.
In this framework, an NGO's impact is then twofold: It's primary impact, sold and attributed to the donor, and it's secondary impact on the life satisfaction of the donor, attributed to the employees and other cost drivers.
This approach is not a substitute for measuring any impact beyond life satisfaction of consumers. It cannot replace studies on the primary impact of NGOs or impact on factors beyond life satisfaction, like the environment.
You can use this framework to compare the impact of companies when deciding between jobs, to compare the impact of donations to the impact from a job, or use it as a guide on how to attribute the impact of a for-profit or NGO to it's customers, employees and suppliers.
The corresponding spreadsheet can be used to apply this framework to any company using commonly available data.
Idea: Using money as a proxy for impact
Impact estimation is a dark art: Complex spreadsheets with high sensitivity to parameters, convoluted theories of change, cherry-picked KPIs that can't be compared between two organizations, triple-counting of the same impact for multiple stakeholders, missing studies to back up assumptions. I am working for a social enterprise and in the five years I've been here I tried to estimate impact a couple of times and always gave up frustrated.
In this article I am trying to turn impact estimation around: Instead of trying to estimate impact from detailed bottom-up models, I will take a top-down approach and work with one measure that works for almost anything: money. At its heart, money represents value delivered and I want to see how far we can take this to estimate impact.
I will guide you through this in three steps that you can also follow and adjust in
this spreadsheet: We will start with estimating the impact of a donation to GiveDirectly. From this we will derive income-weighted revenue as a proxy measure for impact. With this we can then estimate the impact of a market stand as a simple example and then generalize to any organization[1], any job, and the impact of shareholders. Finally we'll try to apply this approach to a few examples, look at the extremes, and see how everything holds up.
The impact of a cash transfer
Let's start with estimating the impact of a donation to GiveDirectly. Most readers are probably familiar with the relation between income and life satisfaction: Life satisfaction is highly correlated to the logarithm of income - at low income levels life satisfaction grows fast with income while at high income levels extra income will have almost no effect on life satisfaction. This holds both when compared between countries and for different income levels within countries.
Our World In Data for more details on this.
We can use this relationship to estimate the impact of donating to GiveDirectly: Let's say we are looking at a pool of donors whose average yearly income is $50k. That would mean they have, on average, a life satisfaction of about 6.9 Life Satisfaction Points (LSP). Then when they donate ten percent of thei...
TL;DR
Building on the logarithmic relationship between income and life satisfaction, we model the purchase of goods as an exchange of money for life satisfaction.
Under that lens, a company's impact on the life satisfaction of its customers is the income-weighted sum of the company's revenue.
By comparing the company's revenue with its cost structure, we can attribute the relative share of impact to the company's employees, suppliers, and shareholders.
While this approach primarily focuses on for-profit companies, it can be extended to NGOs by reframing them as companies selling attribution of their primary impact for donations.
In this framework, an NGO's impact is then twofold: It's primary impact, sold and attributed to the donor, and it's secondary impact on the life satisfaction of the donor, attributed to the employees and other cost drivers.
This approach is not a substitute for measuring any impact beyond life satisfaction of consumers. It cannot replace studies on the primary impact of NGOs or impact on factors beyond life satisfaction, like the environment.
You can use this framework to compare the impact of companies when deciding between jobs, to compare the impact of donations to the impact from a job, or use it as a guide on how to attribute the impact of a for-profit or NGO to it's customers, employees and suppliers.
The corresponding spreadsheet can be used to apply this framework to any company using commonly available data.
Idea: Using money as a proxy for impact
Impact estimation is a dark art: Complex spreadsheets with high sensitivity to parameters, convoluted theories of change, cherry-picked KPIs that can't be compared between two organizations, triple-counting of the same impact for multiple stakeholders, missing studies to back up assumptions. I am working for a social enterprise and in the five years I've been here I tried to estimate impact a couple of times and always gave up frustrated.
In this article I am trying to turn impact estimation around: Instead of trying to estimate impact from detailed bottom-up models, I will take a top-down approach and work with one measure that works for almost anything: money. At its heart, money represents value delivered and I want to see how far we can take this to estimate impact.
I will guide you through this in three steps that you can also follow and adjust in
this spreadsheet: We will start with estimating the impact of a donation to GiveDirectly. From this we will derive income-weighted revenue as a proxy measure for impact. With this we can then estimate the impact of a market stand as a simple example and then generalize to any organization[1], any job, and the impact of shareholders. Finally we'll try to apply this approach to a few examples, look at the extremes, and see how everything holds up.
The impact of a cash transfer
Let's start with estimating the impact of a donation to GiveDirectly. Most readers are probably familiar with the relation between income and life satisfaction: Life satisfaction is highly correlated to the logarithm of income - at low income levels life satisfaction grows fast with income while at high income levels extra income will have almost no effect on life satisfaction. This holds both when compared between countries and for different income levels within countries.
Our World In Data for more details on this.
We can use this relationship to estimate the impact of donating to GiveDirectly: Let's say we are looking at a pool of donors whose average yearly income is $50k. That would mean they have, on average, a life satisfaction of about 6.9 Life Satisfaction Points (LSP). Then when they donate ten percent of thei...