The Nonlinear Library EA - Incorporating and visualizing uncertainty in cost effectiveness analyses: A walkthrough using GiveWell's estimates for StrongMinds by Jamie Elsey
Nov 7, 2023
30:49
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Incorporating and visualizing uncertainty in cost effectiveness analyses: A walkthrough using GiveWell's estimates for StrongMinds, published by Jamie Elsey on November 7, 2023 on The Effective Altruism Forum.
A common first step towards incorporating uncertainty into a cost effectiveness analysis (CEA) is to express not just a point estimate (i.e., a single number) for an input to the CEA, but to provide some indicator of uncertainty around that estimate. This might be termed an optimistic vs. pessimistic scenario, or as the lower and upper bounds of some confidence or uncertainty interval. A CEA is then performed by combining all of the optimistic inputs to create an optimistic final output, and all
of the pessimistic inputs to create a final pessimistic output. I refer to this as an 'interval-based approach'. This can be contrasted with a fuller 'probabilistic approach', in which uncertainty is defined through the use of probabilistic distributions of values, which represent the range of possibilities we believe the different inputs can take. While many people know that a probabilistic approach circumvents shortcomings of an interval-based approach, they may not know where to even begin in
terms of what different distributions are possible, or the kinds of values they denote.
I hope to address this in the current post and the accompanying application. Concretely, I aim to:
Show how performing a CEA just using an interval-based approach can lead to a substantial overestimation of the uncertainty implied by one's initial inputs, and how using a probabilistic approach can correct this while also enabling additional insights and assessments
Introduce a new tool I have developed - called Distributr - that allows users to get more familiar and comfortable with a range of different distributions and the kinds of values they imply
Use this tool to help generate a probabilistic approximation of the inputs GiveWell used in their assessment of Strongminds,[1] and perform a fuller probabilistic assessment based upon these inputs
Show how this can be done without needing to code, using Distributr and a simple spreadsheet
I ultimately hope to help the reader to feel more capable and confident in the possibility of incorporating uncertainty into their own cost effectiveness analyses.
Propagating uncertainty and the value of moving beyond an interval-based approach
Cost effectiveness analysis involves coming up with a model of how various different factors come together to determine both how effective some intervention is, and the costs of its delivery. For example, when we think about distributing bed nets for malaria prevention, we might consider how the cost of delivery can vary across different regions, how the effects of bed net delivery will depend on the likelihood that people use the bed nets for their intended purpose, and the probability that
recipients will install the bed nets properly. These and other factors all come together to produce an estimate of the cost effectiveness of an intervention, which will depend on the values we ascribe to the various inputs.
One way that a researcher might seek to express uncertainty in these inputs is by placing reasonable upper and lower bounds on their estimates for each of them. The researcher might then seek to propagate this uncertainty in the inputs into the anticipated uncertainty in the outputs by performing the same cost effectiveness calculations as they did on their point estimates on their upper bounds, and on their lower bounds, thereby producing corresponding upper and lower bounds on the final cost effectiveness.
An example of an interval-based approach is GiveWell's
assessment of Happier Lives Institute's CEA for StrongMinds. The purpose of this post is not to provide an independent evaluation of Strongminds, nor i...
A common first step towards incorporating uncertainty into a cost effectiveness analysis (CEA) is to express not just a point estimate (i.e., a single number) for an input to the CEA, but to provide some indicator of uncertainty around that estimate. This might be termed an optimistic vs. pessimistic scenario, or as the lower and upper bounds of some confidence or uncertainty interval. A CEA is then performed by combining all of the optimistic inputs to create an optimistic final output, and all
of the pessimistic inputs to create a final pessimistic output. I refer to this as an 'interval-based approach'. This can be contrasted with a fuller 'probabilistic approach', in which uncertainty is defined through the use of probabilistic distributions of values, which represent the range of possibilities we believe the different inputs can take. While many people know that a probabilistic approach circumvents shortcomings of an interval-based approach, they may not know where to even begin in
terms of what different distributions are possible, or the kinds of values they denote.
I hope to address this in the current post and the accompanying application. Concretely, I aim to:
Show how performing a CEA just using an interval-based approach can lead to a substantial overestimation of the uncertainty implied by one's initial inputs, and how using a probabilistic approach can correct this while also enabling additional insights and assessments
Introduce a new tool I have developed - called Distributr - that allows users to get more familiar and comfortable with a range of different distributions and the kinds of values they imply
Use this tool to help generate a probabilistic approximation of the inputs GiveWell used in their assessment of Strongminds,[1] and perform a fuller probabilistic assessment based upon these inputs
Show how this can be done without needing to code, using Distributr and a simple spreadsheet
I ultimately hope to help the reader to feel more capable and confident in the possibility of incorporating uncertainty into their own cost effectiveness analyses.
Propagating uncertainty and the value of moving beyond an interval-based approach
Cost effectiveness analysis involves coming up with a model of how various different factors come together to determine both how effective some intervention is, and the costs of its delivery. For example, when we think about distributing bed nets for malaria prevention, we might consider how the cost of delivery can vary across different regions, how the effects of bed net delivery will depend on the likelihood that people use the bed nets for their intended purpose, and the probability that
recipients will install the bed nets properly. These and other factors all come together to produce an estimate of the cost effectiveness of an intervention, which will depend on the values we ascribe to the various inputs.
One way that a researcher might seek to express uncertainty in these inputs is by placing reasonable upper and lower bounds on their estimates for each of them. The researcher might then seek to propagate this uncertainty in the inputs into the anticipated uncertainty in the outputs by performing the same cost effectiveness calculations as they did on their point estimates on their upper bounds, and on their lower bounds, thereby producing corresponding upper and lower bounds on the final cost effectiveness.
An example of an interval-based approach is GiveWell's
assessment of Happier Lives Institute's CEA for StrongMinds. The purpose of this post is not to provide an independent evaluation of Strongminds, nor i...