The Nonlinear Library EA - How We Plan to Approach Uncertainty in Our Cost-Effectiveness Models by GiveWell
Jan 3, 2024
46:53
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How We Plan to Approach Uncertainty in Our Cost-Effectiveness Models, published by GiveWell on January 3, 2024 on The Effective Altruism Forum.
Author: Adam Salisbury, Senior Research Associate
Summary
In a nutshell
We've received criticism from multiple sources that we should model uncertainty more explicitly in our cost-effectiveness analyses. These critics argue that modeling uncertainty, via Monte Carlos or other approaches, would keep us from being fooled by the optimizer's curse[1] and have other benefits.
Our takeaways:
We think we're mostly addressing the optimizer's curse already by skeptically adjusting key model inputs, rather than taking data at face value. However, that's not always true, and we plan to take steps to ensure we're doing this more consistently.
We also plan to make sensitivity checks on our parameters and on bottom-line cost-effectiveness a more routine part of our research. We think this will help surface potential errors in our models and have other transparency and diagnostics benefits.
Stepping back, we think taking uncertainty more seriously in our work means considering perspectives beyond our model, rather than investing more in modeling. This includes factoring in external sources of evidence and sense checks, expert opinion, historical track records, and qualitative features of organizations.
Ways we could be wrong:
We don't know if our parameter adjustments and approach to addressing the optimizer's curse are correct. Answering this question would require comparing our best guesses to "true" values for parameters, which we typically don't observe.
Though we think there are good reasons to consider outside-the-model perspectives, we don't have a fully formed view of how to bring qualitative arguments to bear across programs in a consistent way. We expect to consider this further as a team.
What is the criticism we've received?
In our cost-effectiveness analyses, we typically do not publish uncertainty analyses that show how sensitive our models are to specific parameters or uncertainty ranges on our bottom line cost-effectiveness estimates. We've received multiple critiques of this approach:
Noah Haber argues that, by not modeling uncertainty explicitly, we are subject to the optimizer's curse. If we take noisy effect sizes, burden, or cost estimates at face value, then the programs that make it over our cost-effectiveness threshold will be those that got lucky draws. In aggregate, this would make us biased toward more uncertain programs. To remedy this, he recommends that (i) we quantify uncertainty in our models by specifying distributions on key parameters and then running Monte Carlo simulations and (ii) we base decisions on a lower bound of the distribution (e.g., the 20th percentile).
Others[2] have argued we're missing out on other benefits that come from specifying uncertainty. By not specifying uncertainty on key parameters or bottom line cost-effectiveness, we may be missing opportunities to prioritize research on the parameters to which our model is most sensitive and to be fully transparent about how uncertain our estimates are. (more)
What do we think about this criticism?
We think we're mostly guarding against the optimizer's curse by skeptically adjusting key inputs in our models, but we have some room for improvement.
The optimizer's curse would be a big problem if we, e.g., took effect sizes from study abstracts or charity costs at face value, plugged them into our models, and then just funded programs that penciled above our cost-effectiveness bar.
We don't think we're doing this. For example, in our vitamin A supplementation cost-effectiveness analysis (CEA), we apply skeptical adjustments to treatment effects to bring them closer to what we consider plausible. In our CEAs more broadly, we triangulate our cost e...
Author: Adam Salisbury, Senior Research Associate
Summary
In a nutshell
We've received criticism from multiple sources that we should model uncertainty more explicitly in our cost-effectiveness analyses. These critics argue that modeling uncertainty, via Monte Carlos or other approaches, would keep us from being fooled by the optimizer's curse[1] and have other benefits.
Our takeaways:
We think we're mostly addressing the optimizer's curse already by skeptically adjusting key model inputs, rather than taking data at face value. However, that's not always true, and we plan to take steps to ensure we're doing this more consistently.
We also plan to make sensitivity checks on our parameters and on bottom-line cost-effectiveness a more routine part of our research. We think this will help surface potential errors in our models and have other transparency and diagnostics benefits.
Stepping back, we think taking uncertainty more seriously in our work means considering perspectives beyond our model, rather than investing more in modeling. This includes factoring in external sources of evidence and sense checks, expert opinion, historical track records, and qualitative features of organizations.
Ways we could be wrong:
We don't know if our parameter adjustments and approach to addressing the optimizer's curse are correct. Answering this question would require comparing our best guesses to "true" values for parameters, which we typically don't observe.
Though we think there are good reasons to consider outside-the-model perspectives, we don't have a fully formed view of how to bring qualitative arguments to bear across programs in a consistent way. We expect to consider this further as a team.
What is the criticism we've received?
In our cost-effectiveness analyses, we typically do not publish uncertainty analyses that show how sensitive our models are to specific parameters or uncertainty ranges on our bottom line cost-effectiveness estimates. We've received multiple critiques of this approach:
Noah Haber argues that, by not modeling uncertainty explicitly, we are subject to the optimizer's curse. If we take noisy effect sizes, burden, or cost estimates at face value, then the programs that make it over our cost-effectiveness threshold will be those that got lucky draws. In aggregate, this would make us biased toward more uncertain programs. To remedy this, he recommends that (i) we quantify uncertainty in our models by specifying distributions on key parameters and then running Monte Carlo simulations and (ii) we base decisions on a lower bound of the distribution (e.g., the 20th percentile).
Others[2] have argued we're missing out on other benefits that come from specifying uncertainty. By not specifying uncertainty on key parameters or bottom line cost-effectiveness, we may be missing opportunities to prioritize research on the parameters to which our model is most sensitive and to be fully transparent about how uncertain our estimates are. (more)
What do we think about this criticism?
We think we're mostly guarding against the optimizer's curse by skeptically adjusting key inputs in our models, but we have some room for improvement.
The optimizer's curse would be a big problem if we, e.g., took effect sizes from study abstracts or charity costs at face value, plugged them into our models, and then just funded programs that penciled above our cost-effectiveness bar.
We don't think we're doing this. For example, in our vitamin A supplementation cost-effectiveness analysis (CEA), we apply skeptical adjustments to treatment effects to bring them closer to what we consider plausible. In our CEAs more broadly, we triangulate our cost e...