The Limits of Nonprofit Impact: Examining A Recent Scholarly Analysis
Posted on 15 July 2010
By Sherine Jayawickrama
A recent Harvard Business School working paper titled The Limits of Nonprofit Impact: A Contingency Framework for Measuring Social Performance (by Alnoor Ebrahim and V. Kasturi Rangan) comes to an interesting conclusion: that measuring impact only makes sense under a limited set of circumstances.
In their May 2010 paper, Ebrahim and Rangan consider the debates on performance and impact of nonprofit organizations playing out in private foundations, U.S. nonprofits and international development actors (both donor agencies and NGOs). They observe that there has been increasing pressure on NGOs to measure performance at all levels of the logic chain: from inputs, activities and outputs (that are under their control) to broader outcomes and impacts (that they have little control over).
Their review of evaluation literature finds that the term “impact evaluation” often refers to evaluations involving a counterfactual (what would have happened without the intervention being evaluated) and that experimental designs using randomized control trials (RCTs) are now considered the “gold standard” for assessing impact.
Ebrahim and Rangan also acknowledge the literature challenging the notion of one “gold standard.” RCTs are often unsuitable for evaluating work that seeks to influence complex development pathways which involve multiple, interacting and non-linear causal factors. The authors recognize the variety of new approaches to evaluation that try to combine methodological rigor with the ability to adapt to complex, non-linear systems.
Against this backdrop, Ebrahim and Rangan look at the variety of work that NGOs engage in (emergency and relief work, service delivery work of small vs large scale/scope, and advocacy and rights-based work) and explore how far down the logic chain results can reasonably be measured. They find that, except in the case of service delivery at significant scale and scope, it is not feasible for NGOs to measure performance beyond outputs and outcomes. This presents a dilemma, especially for funders who increasingly seek evidence of impact.
In response, Ebrahim and Rangan offer a contingency framework based on the notion that “what you should measure is contingent on what you’re trying to achieve.” Their framework is built around two dimensions: an intervention’s theory of change (the causal logic underlying the intervention) and its operational strategy (how an organization implements interventions that advance its mission). They use this framework to define four types of results, along with appropriate measures.
- Niche Results – these results can be expected when theory of change and operational strategy are both focused (e.g. soup kitchens) and appropriate measures would be inputs, activities and outputs.
- Integrated Results – these results can be expected when theory of change is focused but operational strategy is complex (i.e. humanitarian relief, health services) and appropriate measures would be aggregate outputs, outcomes and sometimes impacts.
- Institutional Results – these results can be expected when theory of change is complex but operational strategy is focused (e.g. policy advocacy) and appropriate measures would be outputs and “influence” (or intermediate outcomes).
- Ecosystem Results – these results can be expected when theory of change and operational strategy are both complex (e.g. rights-based approaches to development) and appropriate measures would be outcomes and impacts (over a timeframe in which systemic changes could reasonably have taken place).
Of course, the work of NGOs does not fall neatly into one of these four quadrants and the authors acknowledge that. Their intent is to move beyond the rather polarized debate on impact measurement – between those who want impact to be measured quantifiably (and equate that to rigor and accountability) and those who argue that development work is so context-specific that it is very hard to measure – to a conversation about where on the logic chain measurement should focus.
I found the paper interesting in that it challenges the notion that NGOs must demonstrate impact in order to prove that they are effective and accountable. The authors are realistic about how little control NGOs and their funders have over factors that either perpetuate social problems or contribute to their resolution; at the same time, they don’t let anyone off the hook. Ebrahim and Rangan urge NGOs to measure results as far down the logic chain as reasonable, but they warn against going too far. That would lead to wasted resources, diverted attention and loss of credibility.
The paper spurred some questions for me. First, the most innovative NGO efforts cut across two or more quadrants that Ebrahim and Rangan identify (combining service delivery with capacity building, empowerment, policy advocacy, etc.). To the extent that this work exemplifies complex theories of change and operational strategies, the contingency framework warrants measurement of outcomes and impact. Yet, these efforts often work within messy, non-linear systems over which NGOs have little control. While an NGO’s contribution to progress can be evaluated, the attribution of results is problematic. What types of measures (and methods) would be most valuable for learning, most credible for upward accountability and most meaningful for downward accountability?
Second, having a focused theory of change in a complex setting (e.g. emergency response operations in the wake of a natural disaster or in the midst of a conflict) may be necessary to drive implementation forward efficiently, but it might paper over social and political dynamics with which NGO interventions inevitably interact. Does the contingency framework adequately address the importance of assessing unintended outcomes?
Finally, although there is much I like about this framework and its implications, it feels a bit distant from the main protagonists of development and social change: members of poor communities, their aspirations, their agency and their rights. Management terminology and development jargon often have that effect! How would this framework be different if NGO accountability were looking toward communities rather than (or in addition to) funders?
Sherine Jayawickrama manages the Humanitarian & Development NGOs domain of practice – and the Humanitarian & Development NGOs blog – at the Hauser Center for Nonprofit Organizations at Harvard University.
4 responses to The Limits of Nonprofit Impact: Examining A Recent Scholarly Analysis
Leave a Response
Subscribe
Sherine – your final question is very well put. The biggest evaluation gap is that impoverished communities lack basic information about their own situation: nutrition rates, drop-out rates, crop yields etc. With this information, an organized community will take action. NGOs can provide capacity building and advocacy for communities to fulfill this basic right to information – and then “impact evaluation” will be a natural and in-built result.
Sherine,
First, thank you for a terrific and concise summary of our paper, and especially for your thoughtful questions about it.
Your first question focuses on NGOs that aim for ecosystem-level results (where causality is complex, as are operational strategies). In this context, you ask what kinds of measures and methods would be most valuable for learning, as well as for upward and downward accountability? You have put your finger on precisely the kinds of methodological innovations that are most crucially needed. We need methods that bring together multiple actors (upwards and downwards) in order to gauge their collective impacts, rather than always trying to attribute impacts to single interventions or organizations.
There are a number of examples of such methods in international development. Outcome mapping, developed by the IDRC, typically brings together multiple players — NGOs, policymakers, communities, and other change agents— for an iterative planning process. Its main aim is to change institutional relationships that can facilitate better performance among all actors. Interestingly, outcome mapping does not focus on impacts; it focuses on changing behaviors in relationships at an ecosystem level. As another example, your recent blog post on “Evaluating Advocacy” points to a toolkit developed by iScale, which also involves multiple actors and complex theories of change. We need more innovations like this since, ultimately, ecosystem level problems need ecosystem level methodologies.
In terms of your question about addressing unintended outcomes, it depends on how one uses the contingency framework. We developed the framework as a way for NGOs to think about their intended results, and to link these goals to their theories of change and their operations. If you can get some clarity about what you expect to see from your work, then you are also better positioned to notice things that were unintended or surprising. Otherwise there is a temptation to take credit for all positive results (intended or not), and to sweep the rest under the rug.
On the other hand, I would hesitate to use the contingency framework for the explicit purpose of identifying unintended results. There are better methodologies for doing that, such as the remarkably simple “Most Significant Changes” technique which has been used very effectively for capturing results, intended and unintended.
Finally, you and John both note that the framework feels a bit distant from the main protagonists of development, particularly members of poor communities and their rights. The framework is targeted primarily to NGOs – to their management, boards, and staff. But its core is the goals and missions of NGOs (as captured in their theories of change), whether those are about rights or basic needs. If the framework is to enable better downward accountability to communities, the onus is on NGOs to be more explicit about what they are trying to achieve and to include communities in setting and assessing those goals. However, I certainly appreciate your point about management terminology and development jargon! It is a reminder that, as scholars, we can do more to make our work more accessible and thus to have greater impact.
Dear All,
This is a very interesting paper. an important resource for our work. I am writing to request a favour from you. In the paper there are mentions to a few tables and images that are not there. Is there a version of this paper available with all the table and images. I would be very grateful for that.
thank you very much
Hussein
Just noticed these tables and images can be found in the end of the paper.
thank you
Hussein