Three Questions to Demonstrate Impact

Last month, my colleague Chris Ellis shared some insight into Pay for Success as part of a larger conversation we’ve been having about innovative financing. Many of our clients are doing innovative work in the public and nonprofit sectors, and have found that thinking creatively about solutions often means facing challenges in securing the necessary resources to implement them. Pay for Success is one such promising model, and it relies heavily on the need to evaluate outcomes – which means that our approach to evaluation needs to be just as thoughtful and innovative as our approach to problem solving.

Evaluation is nothing new to those of you reading this who work in the public or nonprofit sector. Funders require it, donors and constituents want to see it, and firms like ours use it as a tool for clients to dig into what works. What is new, however, is the massive amount of data that is available to the general public. On the one hand, this is great news in terms of new tools at our disposal; on the other hand, more and more funders and decision-makers have come to expect data-driven proof of outcomes and impacts.

In the Pay for Success framework as well as others, randomized control trials (RCTs), which are used in clinical trials, are considered the gold standard of evidence to prove that something is working. In her TED talk, economist Esther Duflo explains why this type of evidence is so valuable in social sectors. The trouble is, there isn’t always time to take on this type of evaluation, and, even if there is, it doesn’t always fit: not all programs and policies can be randomized, nor can control or treatment groups easily be assigned.

Whether you are working in a fast-paced world where the pressure to get things done is as strong as the pressure to prove that what you are doing is effective, or if you’re working on something that doesn’t fit an RCT model, these three questions can help get you on track to start proving what works.

  • How can we show that the cause is related to the effect? The simple way to think this through is the phrase, “if ___, then ___.” Using one of my favorite topics as an example, we could say something like, “If a vacant property becomes occupied, then adjacent property values go up.” If we are already working on trying to increase occupancy, then the task becomes identifying a reasonable way to measure occupancy and to measure property values. We dig deeper into tough measurement questions.
  • How can we show that the cause happened before the effect? This question seems simple, but too often, we fall into a habit of measuring only during and after the program, development, or policy. Sure, maybe property values went up after a vacant property became occupied. But what if property values were already increasing before we started? This is why it’s so important to establish a baseline. Maybe our new question becomes, “How much faster are property values growing around vacant properties that become occupied than around vacant properties that stay vacant?”
  • How can we show that the effect isn’t caused by something else? I am a fan of Tyler Vigen’s Spurious Correlation blog, which sums up this last point. Here is a correlation between honey producing bee colonies in the US and divorce rates in South Carolina:

Surely the sad decline in bee colonies isn’t a win for South Carolina marriages! But as absurd as this example is, unfortunately our evaluations are at risk of adopting the same logic if we don’t ask this last question. Once we can say “If ___, then ___,” we have to then ask the question, “What else could be influencing this pattern?”

Thankfully, there are alternative methods for evaluation that can be used in the place of RCTs while still providing some degree of rigor. Many of them use terms like “natural experiment” and “quasi-experimental design,” which sound scary, expensive, and time-consuming. But at their core, all of these methods center around these three key questions.

You can use these three questions to guide program goals, to get a sense of possible impacts stemming from your work, and to prepare your organization for evaluations in the future. Do you have other questions you find useful to guide evaluations? Let us know!

Professor Kevin Morrison of the University of Pittsburgh Graduate School of Public and International Affairs (GSPIA) shared the linked TED talk during a first-semester statistics course. Thanks for the inspiration!