The Nonlinear Library EA - Getting Started with Impact Evaluation Surveys: A Beginner's Guide by Emily Grundy
Nov 14, 2023
19:39
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Getting Started with Impact Evaluation Surveys: A Beginner's Guide, published by Emily Grundy on November 14, 2023 on The Effective Altruism Forum.
In 2023, I provided research consulting services to help AI Safety Support evaluate their organisation's impact through a survey[1]. This post outlines a) why you might evaluate impact through a survey and b) the process I followed to do this. Reach out to myself or Ready Research if you'd like more insight on this process, or are interested in collaborating on something similar.
Epistemic status
This process is based on researching impact evaluation approaches and theory of change, reviewing what other organisations do, and extensive applied academic research and research consulting experience, including with online surveys (e.g.,
the SCRUB study). I would not call myself an impact evaluation expert, but thought outlining my approach could still be useful for others.
Who should read this?
Individuals / organisations whose work aims to impact other people, and who want to evaluate that impact, potentially through a survey.
Examples of those who may find it useful include:
A career coach who wants to understand their impact on coachees;
A university group that runs fellowship programs, and wants to know whether their curriculum and delivery is resulting in desired outcomes;
An author who has produced a blog post or article, and wants to assess how it affected key audiences.
Summary
Evaluating the impact of your work can help determine whether you're actually doing any good, inform strategic decisions, and attract funding. Surveys are sometimes (but not always) a good way to do this.
The broad steps I suggest to create an impact evaluation survey are:
Articulate what you offer (i.e., your 'services'): What do you do?
Understand your theory of change: What impact do you hope it has, and how?
Narrow in on the survey: How can a survey assess that impact?
Develop survey items: What does the survey look like?
Program and pilot the survey: Is the survey ready for data collection?
Disseminate the survey: How do you collect data?
Analyse and report survey data: How do you make sense of the results?
Act on survey insights: What do you do about the results?
Why conduct an impact evaluation survey?
There are two components to this: 1) why evaluate impact and 2) why use a survey to do it.
Why evaluate impact?
This is pretty obvious: to determine whether you're doing good (or, at least, not doing bad), and how much good you're doing. Impact evaluation can be used to:
Inform strategic decisions. Collecting data can help you decide whether doing something (e.g., delivering a talk, running a course) is worth your time, or what you should do more or less of.
Attract funding. Being able to demonstrate (ideally good) impact to funders can strengthen applications and increase sustainability.
Impact evaluation is not just about assessing whether you're achieving your desired outcomes. It can also involve understanding why you're achieving those outcomes, and evaluating different aspects of your process and delivery. For example, can people access your service? Do they feel comfortable throughout the process? Do your services work the way you expect them to?
Why use a survey to evaluate impact?
There are several advantages of using surveys to evaluate impact:
They are relatively low effort (e.g., compared to interviews);
They can be easily replicated: you can design and program a survey that can be used many times over (either by you again, or by others);
They can have a broad reach, and are low effort for participants to complete (which means you'll get more responses);
They are structured and standardised, so it can be easier to analyse and compare data;
They are very scalable, allowing you to collect data from hundreds or thousands of respond...
In 2023, I provided research consulting services to help AI Safety Support evaluate their organisation's impact through a survey[1]. This post outlines a) why you might evaluate impact through a survey and b) the process I followed to do this. Reach out to myself or Ready Research if you'd like more insight on this process, or are interested in collaborating on something similar.
Epistemic status
This process is based on researching impact evaluation approaches and theory of change, reviewing what other organisations do, and extensive applied academic research and research consulting experience, including with online surveys (e.g.,
the SCRUB study). I would not call myself an impact evaluation expert, but thought outlining my approach could still be useful for others.
Who should read this?
Individuals / organisations whose work aims to impact other people, and who want to evaluate that impact, potentially through a survey.
Examples of those who may find it useful include:
A career coach who wants to understand their impact on coachees;
A university group that runs fellowship programs, and wants to know whether their curriculum and delivery is resulting in desired outcomes;
An author who has produced a blog post or article, and wants to assess how it affected key audiences.
Summary
Evaluating the impact of your work can help determine whether you're actually doing any good, inform strategic decisions, and attract funding. Surveys are sometimes (but not always) a good way to do this.
The broad steps I suggest to create an impact evaluation survey are:
Articulate what you offer (i.e., your 'services'): What do you do?
Understand your theory of change: What impact do you hope it has, and how?
Narrow in on the survey: How can a survey assess that impact?
Develop survey items: What does the survey look like?
Program and pilot the survey: Is the survey ready for data collection?
Disseminate the survey: How do you collect data?
Analyse and report survey data: How do you make sense of the results?
Act on survey insights: What do you do about the results?
Why conduct an impact evaluation survey?
There are two components to this: 1) why evaluate impact and 2) why use a survey to do it.
Why evaluate impact?
This is pretty obvious: to determine whether you're doing good (or, at least, not doing bad), and how much good you're doing. Impact evaluation can be used to:
Inform strategic decisions. Collecting data can help you decide whether doing something (e.g., delivering a talk, running a course) is worth your time, or what you should do more or less of.
Attract funding. Being able to demonstrate (ideally good) impact to funders can strengthen applications and increase sustainability.
Impact evaluation is not just about assessing whether you're achieving your desired outcomes. It can also involve understanding why you're achieving those outcomes, and evaluating different aspects of your process and delivery. For example, can people access your service? Do they feel comfortable throughout the process? Do your services work the way you expect them to?
Why use a survey to evaluate impact?
There are several advantages of using surveys to evaluate impact:
They are relatively low effort (e.g., compared to interviews);
They can be easily replicated: you can design and program a survey that can be used many times over (either by you again, or by others);
They can have a broad reach, and are low effort for participants to complete (which means you'll get more responses);
They are structured and standardised, so it can be easier to analyse and compare data;
They are very scalable, allowing you to collect data from hundreds or thousands of respond...