Measure What Matters: How to get started with outcome metrics
Outcome metrics are one of the most important aspects of program management because let your organization know if you’re actually achieving the goals stated in your mission. In this article, we’ll explore what outcome metrics are — and equally as important — what they aren’t.
Prefer to watch instead of read? Check out the video below.
What are outcome metrics?
Outcome metrics are benefits or behavioral changes that can be causally linked to a particular program, such as a decrease in the percentage of unhoused individuals returning to a homeless situation or an increase in the number of children who have access to three meals a day. They’re the big, hairy goals your organization is working toward by using its inputs, such as labor, donations, and grants, to do activities that produce outputs, such as people served, classes taught, and performances held.
It’s important to understand that outputs and outcomes are not the same.
- Outputs are the results of your organization’s day-to-day activities.
- Outcomes are the goals your organization is trying to achieve.
In the case of a homeless shelter, outputs are meals served, individuals sheltered, and cases managed. But that shelter doesn’t exist to serve meals, provide beds, and hold case management meetings. It exists to permanently house unhoused individuals — the outcome. At the end of the day, that is what donors, grant funders, Board members, and other stakeholders care about. Sure, the outputs are critically important because, without them, you would never actually achieve your outcomes. But focusing only on outputs means you can’t measure whether your organization is truly successful.
How do you measure outcomes?
Step 1: Figure out what matters
Gather a group of internal and external stakeholders, such as management, the Board, customers, and similar organizations who each have unique perspectives on your programs and the role that outcome metrics will play in your organization. Your organization exists to serve its customers — the people receiving services — so you want to be sure that you’re balancing what they care about with what your organization cares about.
Step 2: Choose the outcomes that you want to measure
Once you’ve established what is important to your organization, you need to decide which programs you want to measure, the appropriate outcomes with which to measure them, and the measurement period. If you’re just starting out with outcome metrics, it’s a good idea to focus on 1 program and 2–5 outcome metrics.
There are three major types of outcomes that you can choose.
- initial outcomes — the first benefits or changes that program participants experience
- intermediate outcomes — longer-term outcomes that show progress toward the final goal
- long-term outcomes — the ultimate outcomes a program is trying to achieve.
Let’s look at an organization whose long-term outcome is to improve high school graduation rates: There’s a lot of time between Kindergarten and graduation, so the organization needs to choose outcomes that show incremental progress toward its long-term outcome. The organization’s initial outcome measures literacy rates since literacy is directly linked to graduation rates. As the student gets older, the organization measures intermediate goals like attendance, honor roll, and standardized test scores, culminating in the long-term outcome of the graduation rate.
Step 3: Specify indicators for your outcomes
Now that you’ve specified the outcomes you want to measure, you have to specify your outcome indicators, or the factors that have a significant impact on your organization’s ability to achieve — or not achieve — an outcome. These can be internal or external to your program and they may or may not be within your control. Remember that saying from school that correlation doesn’t mean causation? That’s a fancy way of saying that just because someone is using your organization’s service(s) doesn’t mean that service(s) is causing — or not causing — the individual’s behavior differences — or lack thereof.
For example, let’s say you run a drug rehab program. You have two patients: one who is showing massive improvement and one who isn’t. You could conclude that your program is amazing and that the patient who isn’t showing improvement is just a hard case, or you could conclude that your program is insufficient and the patient showing massive improvement is just highly motivated. In reality, there are a ton of other factors, such as a robust support system, number of years of drug use, proximity to other drug users, whether or not the patient has a job, dual diagnosis, and so on. It’s imperative to identify as many of these factors as possible because without accounting for them in your outcome metrics, you don’t actually know if your program is responsible for the outcomes.
Step 4: Collect data.
Now that the preparation work is done, it’s time to shift focus to data collection. You need to identify data collection methods that are cost-effective but still result in complete, useful, and credible data. Then, standardize the data collection procedures so you don’t accidentally skew your data.
Let’s look at an example: When we were first designing Grantcycle, we interviewed dozens of nonprofits and grant consultants. In order to compare their responses, we needed to standardize the questions asked during each interview. In addition, we used open-ended questions (“What would help solve your problem?”) rather than leading questions (“Would feature X solve your problem?”) so we didn’t accidentally sway their answers. Whatever data collection method you choose, make sure that every touch point is as uniform as possible so your data is as reliable as possible.
Step 5: Deploy your outcome measurement system
Now you get to actually see your outcome metric system at work. If this is your first time measuring outcomes, it’s a good idea to pilot it with a small test sample. This can help you avoid deploying something that has major flaws that weren’t evident previously. For example, let’s say an orchestra wants concert attendees to fill out paper surveys and turn them in as they leave the venue, but no one picks up the papers or they fall out of attendees’ playbills, or they’re just thrown away. Obviously, this is not going to result in quality data from a large group of respondents. At this point, the only option is to go back to the drawing board and redesign the collection methods because, at the end of the day, your outcome metrics are only as good as your ability to measure them.
Step 6: Analyze and report your findings
To analyze your data, enter the raw data into data analytics software, check for errors, and then analyze the data by key characteristics. The platform you use doesn’t have to be fancy like Microsoft Power BI or Tableau. For most purposes, platforms like Google Sheets, Microsoft Excel, or Airtable will be sufficient, won’t cost much money, and won’t require extensive training. Regardless of your tools, keep in mind that you can make data say whatever you want to, so this step requires integrity and possibly a quick statistics refresher. Next, you need to explain your findings in a narrative and graphical format taking into account the outcome indicators that are or are not within your control. The goals are to explain why you did or didn’t achieve an outcome, indicate plausible reasons for the outcome based on your indicators, and provide context for the results within your organization and industry at large.
Step 7: Improve your system
If you were running a pilot, this is the part where you take your learnings and tweak your outcome measurement system. If you were already involved in full-scale implementation, make sure you’re being sensitive to changes inside and outside your organization and the effect that those changes have on your measurement system. What works for one measurement cycle may not work forever. If you do change your system, make sure that you keep a record of those changes and why. Any changes will alter the relevance of period-over-period comparison, and you want to make sure that those variables are disclosed to stakeholders.
Step 8: Use your findings
Now it’s time to use the data gleaned from measuring your outcome metrics to provide direction for your staff, identify training and development needs, improve programs, build budgets, and guide your fundraising and grant strategies.
Measuring outcomes is no small task. It requires calculated planning, design, and implementation that can take months of preparation and requires a coordinated effort across your organization. That said, the time and effort are worth it. Outcome metric data will not only make your organization more effective, but also expand opportunities for collaboration, increase donations, and make your organization more appealing to grant funders.
Grantcycle’s grant management software helps nonprofits, for-profits, and government entities plan, measure, and articulate their impact in real time. To learn more, go to grantcycle.com.
