This is the first in a series of blog posts on the work and findings of the Listening Project, which organized over 20 Listening Exercises in various contexts and regions since late 2005.  The Listening Project is a systematic exploration of the insights of people who live in societies receiving international assistance (humanitarian assistance, development cooperation, peace-building activities, human rights work and environmental conservation).   More than 130 international and local organizations contributed over 400 staff members to Listening Teams that held conversations with nearly 6,000 people over the last 5 years. 

By Dayna Brown

The Listening Project has listened to the experiences and reflections of a wide range of local people (and not just “key stakeholders”), including aid recipients, community members and leaders, government officials, civil society and religious representatives, teachers, health workers, business people, academics, NGO and CBO staff, women, and youth. 

Each Listening Exercise produced a report that captures in rich detail the stories, opinions and perspectives of local people on the cumulative effects of international assistance on their lives and their societies.  The Listening Project is now analyzing the evidence from these conversations and is writing Issue Papers which highlight some of the common concerns that were raised by people across these locations.

What has been most striking to us is that how people experience international assistance and the system that they describe is remarkably similar across geographical areas and contexts. 

While donors and aid agencies have committed to involving aid recipients more and to improving accountability (through the Paris Declaration on Aid Effectiveness, the Accra Agenda for Action, the Good Humanitarian Donorship Initiative, the Humanitarian Accountability Partnership, etc.), we found that most donors and aid agencies do not spend much time listening to local people’s perspectives or reflecting on the impacts of their work, much less the cumulative effects of their and others’ interventions. Several people in different places said, “no one has ever asked us our opinion of aid before this.”  

As the Listening Project analyzed the evidence from these Listening Exercises, we found that the current aid system limits opportunities and incentives for listening in open-ended ways to people on the receiving end of aid efforts.  The head of an INGO on the Thai-Burma border captured the challenges when she said, “Donors demand task focused work. Staff would love to have more time to talk to people in the camp, to spend the night in the camp (which is not allowed).  But we have reports due, with facts, and numbers, and it needs to be right to keep the funding coming. Some NGOs are run like businesses. The donors are not helping us be respectful because they come with their new ideas, trends and we have to jump….We end up with ridiculous time frames to do things. We cut out the process and spend the rest of the year doing damage control.”

While there is increasing discussion on how to improve the effectiveness of aid efforts, the current aid system is still more focused on delivering goods and services efficiently—and this has an effect on the ways agencies and their staff listen, what they listen for, and where, when and to whom they listen. Most agencies listen only to people who are in (not outside of) the chain of delivery and they listen primarily for assessments of efficiency or effectiveness of their projects.

While Listening Teams have heard lots of feedback on specific project details, people everywhere consistently expressed concerns that seemed to go deeper than particular programming flaws.  They say that aid agencies should “invest the necessary time”, “go more slowly”, and “listen to people” in order to “learn about the real circumstances”, “get to know people”, and “show respect for people’s ideas and opinions.” 

People have equated better listening with better outcomes and longer-term impacts. We have much to learn by listening to their ideas about how to improve the effectiveness of international assistance efforts. 

Dayna Brown directs the Listening Project at CDA Collaborative Learning Projects.  She can be contacted at dbrown@cdainc.com.

This is the second of two posts by Steven Lawry exploring why and how developing country governments should set a strong, clear frame for development efforts, and why and how donors and INGOs should work within that frame.

In my last post, I argued that INGOs tend not to work as closely as they should with governments in developing countries.  This has many causes.  An important one is that donors have found INGOs to be reasonably efficient vehicles for channeling funding to developing countries on terms that ensure the donors retain maximum control over program priorities and program management.  

This is short-sighted and undercuts many benefits to development strategy and poverty reduction that would result where both donors and INGOs worked more closely with host governments.  In this post, I offer some reflections on what can be gained by closer donor-INGO-host government management of development strategy and programming.

INGO programs can benefit from the ideas of host-government civil servants

By not engaging actively and routinely with governments, INGOs don’t benefit from the experience and good judgment of local civil servants, many of whom are similarly committed to poverty reduction.  My experience is that local officials possess hard-earned, grassroots experience, and have good ideas on how to execute poverty reduction programs.  The benefit of this knowledge is lost where INGO staff are not obliged to check in regularly with host-country officials, for reviews of program progress and effectiveness, or when these reviews are treated by both sides as perfunctory exercises.

INGOs can help governments implement their development strategies

Some INGO leaders might argue that, as civil society organizations, their role is not to work with governments, but directly with the poorest. I don’t want to argue that INGOs should be directing their staff to work principally with governments or channel their funding through government ministries.  But many developing country governments have development plans and strategies of their own.  INGOs should take cognizance of those plans, and work in concert with the government’s programs where their purposes are consistent with poverty reduction. 

INGO field work should be directed to regions of the country where government sees the greatest needs.  INGOs should make every effort to recruit staff locally to fill positions at the highest levels.  Local staff members not only understand the local context, but tend to be more attuned to host-government plans and priorities than international staff. Surely, good things can happen, in terms of the relevance and sustainability of INGO work and in terms of improved government capacity, if more opportunities were fostered to work together.

Better national policy yields more impactful INGO programming

INGOs tend to operate in a project mode, and not in the realm of policy and program. Sustained poverty reduction requires sympathetic and supportive public policies.  Governments need to join INGOs by directing more of their own resources to the education and health needs of women and girls.  Official banking regulations need to be supportive of micro-finance institutions.  Agricultural pricing policies and land tenure policies need to be supportive of small-holder agriculture. 

Too often, INGOs work in policy environments that have implications counterproductive to the outcomes they promote.  INGOs can help villagers build rural schools and health posts.  But at the end of the day, those facilities are must be staffed by properly trained and appropriately paid teachers and clinicians.  INGOs have an active interest in helping governments get policies right. 

INGOs can help governments manage aid better

A friend who works for USAID here in Juba remarked the other day that some governments, given their many concerns, including the scarcity and shortage of skilled staff, are happy for INGOs to operate more or less freely in providing essential services and needed relief. No doubt this is true.  INGOs often work in very poor countries. The incapacities of host governments are just another expression of that poverty. But the pathway to sustained poverty reduction should also include a strategy that helps governments become better at shaping and managing poverty reduction programs, even under circumstances of dire poverty and limited public finances. 

This is something that President Seretse Khama understood to be a benefit of having expatriates working side by side with Batswana civil servants. When the advisors were gone, a template of good management practice remained. 

Democratic governments will insist on better terms of engagement.

Mention of Seretse Khama and the example of Botswana brings me to the question of what it will take to bring about a new dispensation in donor-INGO-host government relations, one where all three parties work in closer unison on an authentically shared vision for poverty reduction. 

Frankly, the initiative is not going to come from donors or INGOs, despite the obvious advantages of greater collaboration.  Donor governments have their own strongly-held ideas about economic development and poverty reduction, and increasingly have found in INGOs partners willing to work to those ideas. 

The impetus for a new dispensation will likely come from developing country leaders who believe, like Seretse Khama, that their country’s efforts to reduce poverty are too important to be left to donors and INGOs alone, no matter how expert, resourced and good-intentioned they may be. These leaders, like Khama before them, will by definition be democrats, whose sense of urgency for meaningful and rapid progress will be in large part driven by the rising expectations of a citizenry prepared to hold their leaders accountable. 

Steven Lawry, Senior Research Fellow at the Hauser Center for Nonprofit Organizations, is currently based in Juba, Southern Sudan, where he heads a USAID-funded project assisting the Government of Southern Sudan to develop a new land policy.

This is the first of two posts by Steven Lawry exploring why and how developing country governments should set a strong, clear frame for development efforts, and why and how donors and INGOs should work within that frame.

As a young man starting out in international development work in the 1970s, I served as a Peace Corps Volunteer in Botswana.  But my only connection to the Peace Corps was a monthly paycheck (a very small one).  I was for all intents and purposes a Botswana government civil servant.  I held an established government post, Assistant Planner in the Department of Town and Regional Planning. 

The Department was a unit of the Ministry of Local Government and Lands.  I reported to a Senior Planner, a Swede funded by SIDA.  She reported to the Director, who was seconded by UNDP.  He reported to an Undersecretary in the Ministry, who was a Motswana, and so forth. 

I didn’t realize it at the time, but what seemed to be a sensible and effective way of integrating overseas staff into a developing country’s public service was quite unusual.  I came to learn that, in other countries, apart from a few high-level advisors, donor-funded staff usually worked in separate management units, located in aid missions or in the offices of donor-funded contractors and INGOs. 

As I came to work in other countries and under different aid staffing regimes, Botswana’s practice of integrating overseas advisors into its civil service structure seemed all the more compelling.  It helped ensure that everyone worked to a coherent development strategy, which had been debated up and down the structure (by both Batswana and expatriate staff) and agreed ultimately at the highest levels.  overnment programs were backed by donor funds and Botswana government revenues.

Any donor making its funding contingent on activities that fell outside of Botswana’s carefully established priorities would be told, as politely as possible, that they take their funds elsewhere.  Funding was important, yes.  But more important was ensuring that it was applied in a disciplined and coordinated way to the country’s own development strategy.

Donors rarely did take their funds elsewhere, as they came quickly to appreciate the advantages of Botswana’s approach.  Programs were well-conceived and well-executed.  They were scaled to the country’s capacity to actually implement them and enjoyed the full endorsement of the government and the commitment of government staff and financial resources.  They tended to succeed where similar initiatives, pursued in other countries without the benefit of full government endorsement and engagement, failed.   

One obvious lesson of the Botswana model is that operational structure—how donors and donor-supported staff and host governments interact in shaping policies, designing programs and implementing them—really matters. 

I have lived and worked in several other developing countries after my time in Botswana, and kept asking myself and others why the Botswana model of staff integration is so rarely found elsewhere.  Explanations offered, or implied, were various.

One is the nationalist argument: having donor staff so deeply entrenched in national governing institutions would be an affront to national sovereignty.  Moreover, expatriates would be holding positions that should be held by citizens. Many political leaders embrace the nationalist argument, or feel popular pressure to embrace it. 

Botswana’s greatest proponent of its integrated staffing model was the President, Seretse Khama.   Khama was a person of considerable wisdom and self-confidence, and if there were pressures to eschew this approach, he would have tamped them down with reasoned arguments on behalf of its many benefits. 

Importantly, the Botswana model was never meant to be a permanent, open-ended arrangement.  Accompanying heavy use of expatriate staff was a vigorous national education and training program, including major investment in the University of Botswana and overseas post-graduate training.  By the mid-1980s, less than 20 years after Botswana’s independence, the civil service was largely localized. 

Putting the nationalist argument aside, donors have their own reasons for not embracing the model.  One is that, despite efforts to appraise initiatives in light of local conditions, most international donors work to a strategic framework produced in Washington, London, Geneva or Paris.  Donor programming has an adaptation problem: to align their work with the needs of a developing country as defined by its internal planning process would require a tolerance for flexibility and compromise that many aid agencies can’t negotiate with their home governments.

My time in Botswana pre-dated the rise of INGOs.  Since the 1980s, INGOs have come to be major players in international development.  As direct implementers of poverty reduction interventions (especially in the fields of health, education, microfinance, reproductive health, and water and sanitation) they loom particularly large. 

For many INGOs, public funds (including contracts and grants from USAID) are important sources of funding and have allowed them to expand their funding bases significantly beyond private donations.  (In 2009, CARE USA received $240 million in revenue from private sources versus $247 million from USAID.  Also in 2009, World Vision received $344 million from USAID, about 28 percent of its total revenue. In 2008, Save the Children received $108 million, or 24 percent of its revenue, from USAID.)  

INGOs sometimes receive more direct funding from donors than host governments themselves.  Before the January 12 earthquake, 80 percent of all donor funds for Haiti were directed through INGOs; 90 percent of US government funding for Haiti went to INGOs. 

As conduits for funding or providers of advisors and staff, INGOs have even less direct functional engagement with host-country governments than bi-lateral or multi-lateral donors, which (compared to the Botswana standard) is already pretty low. 

By insisting that their money goes directly to contractors and INGOs, donors foster a parallel system of aid management and administration.  INGOs conclude, quite logically, that their principal client is not the host-country government, but the donor.  This encourages many of the dysfunctions and inefficiencies that the Botswana model minimized.

Steven Lawry, Senior Research Fellow at the Hauser Center for Nonprofit Organizations, is currently based in Juba, Southern Sudan, where he heads a USAID-funded project assisting the Government of Southern Sudan to develop a new land policy.

INGOs do not have enough aid resources (money and/or trained personnel) to do everything that they want to do.  They must therefore make difficult decisions about how to allocate their limited resources.  These decisions are shaped to some extent by more or less practical considerations, such as concerns about aid workers’ safety and the need to raise funds.  But INGOs also take moral and ethical considerations into account.  What sorts of moral and ethical considerations should INGOs deem relevant when making distributive decisions?

One consideration that INGOs seem to treat as morally relevant—at least in their public self-presentations— is the number of countries in which they work.  For example:

• “We work in nearly 100 countries to overcome poverty and injustice.”

• ”Doctors Without Borders/Médecins Sans Frontières (MSF) is an international medical humanitarian organization working in more than 60 countries to assist people whose survival is threatened by violence, neglect, or catastrophe.”

• ”Last year, CARE supported more than 800 poverty-fighting projects in 72 countries to reach more than 59 million people.”

• “The International Rescue Committee is on the ground in more than 40 countries, providing emergency relief, relocating refugees, and rebuilding lives in the wake of disaster.” 

Some INGOs argue that working in many different countries is a means to other valuable ends.  For example, Oxfam states (on page 5 of this report) that “with a programme spread across the world, Oxfam has a greater understanding of the many causes of poverty, and we can achieve greater impact.”   The foregoing statements imply that working in many countries has intrinsic moral value.  I am inclined to think that it does not.

An INGO might argue that working in many different countries symbolizes its commitment to the idea that all people are of equal moral worth, regardless of their citizenship, race, nationality, etc.  I see two difficulties with this argument. 

First, providing aid in many different countries can conflict with other important goals that also convey the idea that all people are of equal moral worth, such as alleviating as much severe poverty or acute suffering as possible, providing aid to the worst-off groups, or focusing on victims of intentional injustice. 

Second, the symbolism of working in many different countries is complex.  While an INGO might intend to convey impartiality, it might instead telegraph a desire for power or even domination, by inadvertently invoking colonial patterns.   

What do you think: is it is intrinsically important that emergency and development INGOs work in many different countries? Why or why not?

Jennifer Rubenstein is Assistant Professor of Politics at the University of Virginia and a regular contributor to this blog.

Following this blog’s last several posts on impact measurement (and the discussion that it helped engender), I came across an interesting advocacy toolkit on the iScale website that describes a system that combines planning, monitoring and evaluation – all focused on helping advocates learn and make corrections in realtime.

The toolkit is quite detailed and prescribes a multi-step planning process that is likely to take a lot of time upfront, and it demonstrates just how complex multi-actor advocacy campaigns are and how challenging evaluating progress (let alone results) can be. 

I like that, even as it seeks to build the mindset and infrastructure for evaluation from the outset of an advocacy campaign, the toolkit is realistic about how complex evaluating impact will be. It notes that “in most cases, no single actor, factor or strategy can independently create the change needed to achieve success,” and recognizes that “impact is attained through the combined and coordinated efforts of multiple actors… in conjunction with multiple external factors and conditions.”

Advocacy evaluation is a fairly new field that calls for approaches that can meaningfully tell advocates if they are gaining any traction on – and having any influence over – the policy issues on which they are focused. 

NGOs are increasingly investing in advocacy because they know that progress on the issues they care about – at any significant scale – cannot be made without change in the arenas of policies, laws and institutions. 

But how do they know whether their advocacy efforts make a difference in the broader scheme of things?  How do they know how much to invest in advocacy as compared to field programs or fundraising?  How do they share experience and learning with others advocating on the same issues? 

This new advocacy toolkit offers a glimpse of how the Global AIDS Alliance and its campaign partners are setting about these challenges.

Sherine Jayawickrama manages the Humanitarian & Development NGOs domain of practice – and the Humanitarian & Development NGOs blog – at the Hauser Center for Nonprofit Organizations at Harvard University.

A recent Harvard Business School working paper titled The Limits of Nonprofit Impact: A Contingency Framework for Measuring Social Performance (by Alnoor Ebrahim and V. Kasturi Rangan) comes to an interesting conclusion: that measuring impact only makes sense under a limited set of circumstances.

In their May 2010 paper, Ebrahim and Rangan consider the debates on performance and impact of nonprofit organizations playing out in private foundations, U.S. nonprofits and international development actors (both donor agencies and NGOs).  They observe that there has been increasing pressure on NGOs to measure performance at all levels of the logic chain: from inputs, activities and outputs (that are under their control) to broader outcomes and impacts (that they have little control over). 

Their review of evaluation literature finds that the term “impact evaluation” often refers to evaluations involving a counterfactual (what would have happened without the intervention being evaluated) and that experimental designs using randomized control trials (RCTs) are now considered the “gold standard” for assessing impact. 

Ebrahim and Rangan also acknowledge the literature challenging the notion of one “gold standard.”  RCTs are often unsuitable for evaluating work that seeks to influence complex development pathways which involve multiple, interacting and non-linear causal factors. The authors recognize the variety of new approaches to evaluation that try to combine methodological rigor with the ability to adapt to complex, non-linear systems.

Against this backdrop, Ebrahim and Rangan look at the variety of work that NGOs engage in (emergency and relief work, service delivery work of small vs large scale/scope, and advocacy and rights-based work) and explore how far down the logic chain results can reasonably be measured. They find that, except in the case of service delivery at significant scale and scope, it is not feasible for NGOs to measure performance beyond outputs and outcomes.  This presents a dilemma, especially for funders who increasingly seek evidence of impact.  

In response, Ebrahim and Rangan offer a contingency framework based on the notion that “what you should measure is contingent on what you’re trying to achieve.”  Their framework is built around two dimensions: an intervention’s theory of change (the causal logic underlying the intervention) and its operational strategy (how an organization implements interventions that advance its mission).  They use this framework to define four types of results, along with appropriate measures.

1. Niche Results – these results can be expected when theory of change and operational strategy are both focused (e.g. soup kitchens) and appropriate measures would be inputs, activities and outputs.
2. Integrated Results – these results can be expected when theory of change is focused but operational strategy is complex (i.e. humanitarian relief, health services) and appropriate measures would be aggregate outputs, outcomes and sometimes impacts.
3. Institutional Results – these results can be expected when theory of change is complex but operational strategy is focused (e.g. policy advocacy) and appropriate measures would be outputs and “influence” (or intermediate outcomes).
4. Ecosystem Results – these results can be expected when theory of change and operational strategy are both complex (e.g. rights-based approaches to development) and appropriate measures would be outcomes and impacts (over a timeframe in which systemic changes could reasonably have taken place).

Of course, the work of NGOs does not fall neatly into one of these four quadrants and the authors acknowledge that. Their intent is to move beyond the rather polarized debate on impact measurement – between those who want impact to be measured quantifiably (and equate that to rigor and accountability) and those who argue that development work is so context-specific that it is very hard to measure – to a conversation about where on the logic chain measurement should focus.

I found the paper interesting in that it challenges the notion that NGOs must demonstrate impact in order to prove that they are effective and accountable. The authors are realistic about how little control NGOs and their funders have over factors that either perpetuate social problems or contribute to their resolution; at the same time, they don’t let anyone off the hook.  Ebrahim and Rangan urge NGOs to measure results as far down the logic chain as reasonable, but they warn against going too far. That would lead to wasted resources, diverted attention and loss of credibility. 

The paper spurred some questions for me.  First, the most innovative NGO efforts cut across two or more quadrants that Ebrahim and Rangan identify (combining service delivery with capacity building, empowerment, policy advocacy, etc.). To the extent that this work exemplifies complex theories of change and operational strategies, the contingency framework warrants measurement of outcomes and impact. Yet, these efforts often work within messy, non-linear systems over which NGOs have little control.  While an NGO’s contribution to progress can be evaluated, the attribution of results is problematic.  What types of measures (and methods) would be most valuable for learning, most credible for upward accountability and most meaningful for downward accountability?

Second, having a focused theory of change in a complex setting (e.g. emergency response operations in the wake of a natural disaster or in the midst of a conflict) may be necessary to drive implementation forward efficiently, but it might paper over social and political dynamics with which NGO interventions inevitably interact.  Does the contingency framework adequately address the importance of assessing unintended outcomes?

Finally, although there is much I like about this framework and its implications, it feels a bit distant from the main protagonists of development and social change: members of poor communities, their aspirations, their agency and their rights.  Management terminology and development jargon often have that effect!  How would this framework be different if NGO accountability were looking toward communities rather than (or in addition to) funders? 

Sherine Jayawickrama manages the Humanitarian & Development NGOs domain of practice – and the Humanitarian & Development NGOs blog – at the Hauser Center for Nonprofit Organizations at Harvard University.

Several powerful donors have concluded that nonprofits make inadequate use of impact assessment tools.  They are backing up their arguments with an implicit threat: measure in particular ways or you don’t get the money.  Wise nonprofit leaders know that the problems they work on are not susceptible to simple measurement.  They know that the kind of formal impact measures some donors expect and management consulting firms prescribe are hard to come by honestly.  They collect various data all the time to inform their judgment and decision-making and to spur learning. Now, data collection (to donor-specified standards) is increasingly used for accountability purposes. 

This may have the effect of reducing the degrees of freedom nonprofit leaders have to innovate and to pursue promising but risky ideas (without the fear that failure to prove one idea will poison their chances to learn from that failure and try something else another day).   As former Ford Foundation President Susan Berresford argues, insisting that grantees demonstrate measureable, short-term impact can have the effect of “miniaturizing ambition” for doing risky but potentially break-through work.

People who impose these restrictions confuse use of prescribed tools or achievement of certain outcomes as evidence of good management.  Sometimes they are. But, in and of themselves, they hardly constitute an impressive tool kit of good management practice. 

The good judgment of experienced managers, deeply immersed in the complex social dynamics of the communities in which they work, is a formidable and essential resource in assessing impacts.  Experience and tested judgment also come into play in shaping a picture of the complex variety of social factors that might explain, for instance, why some poor children and not others attend school, or what mix of interventions are most likely to keep kids out of trouble with the police.

Effective nonprofit managers get information from a variety of sources: formal studies, observation of trends in behavior, feedback from partners and clients. They also draw on deep reserves of knowledge of the local social context, of cultural norms and values, and on the ability to empathize, to look at the world through the eyes of others. 

These sources of knowledge are particularly important in shaping untested but potentially innovative, breakthrough approaches to social change. Effective leaders first and foremost seek to explain how a given problem is responding to a given set of interventions.  Data help describe what is happening, but the interpretative powers of managers are essential to meaningful explanation.

One of my favorite examples (see working paper here) of the kinds of insights that arise from observation, judgment and experience is the particular knowledge that Muhammad Yunus gained from walking through poor communities around Chittagong University on his daily walk to work.  His knowledge of rural Bangladeshi society, combined with his advanced training and powers of intuition, spawned his ideas on social lending, or what became known as micro-finance. 

The invention of micro-finance demonstrates that breakthrough innovations, and even simple adjustments to well-established programs, are spawned by a variety of sources and intellectual attributes:  data, data intelligently interpreted, knowledge of the local and comparative contexts, and good judgment.  All four of these factors are essential to shaping development breakthroughs.  Donors should give greater weight to the latter three over the first in considering funding proposals.

A recently published book on the use of applied mathematics to help understand messy, hard-to-measure problems speaks to the importance of experience and judgment in making sense of limited data.  The book is Street-Fighting Mathematics: the Art of Educated Guessing and Opportunistic Problem Solving, by Dr. Sanjoy Mahajan.  Dr Mahajan is associate director of MIT’s Teaching and Learning Laboratory and the book grew out of a course by the same name that Dr. Mahajan taught for several years at MIT.

The basic premise of his approach, set out in the books first sentence, is that “Too much rigor teaches rigor mortis: the fear of making an unjustified leap even when it lands on the correct result.”  Many real-world problems are not easily described with the kind of precision that professional mathematicians insist upon. This is due to the limitations of data, the costs of collecting and analyzing data, and the inherent difficulties of giving mathematical expression to the complexity of human behavior.  
In the face of these obstacles, mathematicians tend to do one of two things: insist on finding the true proof, even in the face of huge methodological constraints (rigor mortis) or give up. 

Mahajan counsels a third-way: using mathematical reasoning to find a good-enough, approximate and usually valid and useful answer; or as Dr. Mahajan so adeptly puts it, “When the going gets tough, the tough lower their standards” (p. 99).  His book describes six tools for better understanding complex problems with limited data, including picture proofs, lumping, and reasoning by analogy.

There is wisdom in Dr. Mahajan’s core argument that is relevant to current debates about the place of impact assessment in program management.  Many problems, especially problems of social analysis, present huge problems of description and accurate measurement.  We can learn much of what we need to know by tracking a few data points, but knowledge of the underlying social forces and personal motivations that frame the decisions people make is essential to specifying what should be measured and interpreting findings wisely.

My concerns about the emphasis some donors give to evaluation and impact assessment lie not in their lack of value, but in a skewing of perspective.  I want to sum up with a few thoughts on getting the perspective in better balance.

• Knowledge of the local context and the insights spawned by that knowledge are hard won and accumulated over many years. External donors and many of their staff too often don’t possess such knowledge.  For large Western donors, reliance on data and impact measures can be a crutch, a substitute for the knowledge of local context they don’t have.  
• Lack of knowledge of context contributes to an overreliance on one-size-fits-all interventions based on experience from elsewhere, resulting in poorly-adapted local project design.   An obvious remedy is to place greater trust in the leadership and judgment of people who live and work close to the problems; local educators, entrepreneurs, civil society leaders.  
• Evaluation is first and foremost a learning tool, of greatest value as an aid to the judgment of program leaders and managers. The work of donors also stands to benefit from the knowledge that grantees gain in assessing changes within the communities they work and progress in pursuing particular goals.
• Of greatest relevance to predicting the merits and eventual success of a proposed grantee initiative are the wisdom, experience, judgment and reputation of the grantee organization and its leadership and staff.   These are the important qualities that should be considered when contemplating a grant.  (William Duggan’s book, Strategic Intuition, examines the qualities of leadership and management that spawn systemic impacts.)
• Donors who insist on short-term measurable impact should stay away from funding work that seeks breakthroughs on complex, long-intractable problems. 

Steven Lawry, Senior Research Fellow at the Hauser Center for Nonprofit Organizations, is currently based in Juba, Southern Sudan, where he heads a USAID-funded project assisting the Government of Southern Sudan to develop a new land policy.

A newly-published essay by Andrew Natsios – who served as USAID Administrator from 2001 to 2006 – lays bare the tension between the compliance side of aid bureaucracies and the programmatic side of those same agencies.  He argues that the balance has now tipped so strongly toward compliance that the integrity of programs is under threat.  He also asserts that the compliance side of aid has taken over management and decision-making at USAID.

It is interesting to see Natsios, who presided over this “counter-bureaucracy” for five years, rip into the compliance culture that he oversaw. He does so with a clarity and insight that should not be ignored as development and foreign assistance policies are being redefined by the Obama Administration and on Capitol Hill.  Ironically, Natsios’ account of these tensions and imbalances may also reveal why these policy processes seem to be so bogged down and delayed.

Natsios provides a historical perspective of how the compliance culture came to be dominant within the U.S. foreign assistance bureaucracy – he calls it “a painful story of good intentions gone bad.”  He is unsparing in his critique of what this has resulted in, and declares that it is well past the point where compliance has become counter-productive. 

Natsios argues that the demands of this compliance culture “are now so intrusive that they have distorted, misdirected and disfigured USAID’s development practice to such a degree that it is compromising U.S. national security objectives and challenging established principles of good development practice.”

The argument that Natsios makes – that increasing pressure to measure outcomes or impacts can lead to a tendency to invest in interventions that can be easily measured – is not a new one, but his “view from the inside” gives even more credence to this perspective. 

Natsios starts from the principle that development programs that are most precisely and easily measured are often the least transformational, and those programs that are most transformational are often the least measurable.  He posits that health programs have become the most favored sector in U.S. foreign assistance because health outcomes are more easily measured, and that democracy and governance programs have been underfunded because their results are hard to measure (especially within short timeframes).

I suspect that Natsios paints this picture so starkly in order to get his central arguments across: that measurability does not equal development significance, that good development must be the unequivocal goal of U.S. foreign assistance and that foreign aid systems and processes must not unintentionally undermine that goal.

Does Natsios go too far?  Although you might conclude that Natsios is targeting measurement as the villain here, I think the point is more nuanced. It is not whether measurement is good or bad.  It matters what measurement is for (e.g. is it for learning and continuously improving development practice, or is it for satisfying compliance requirements?) and how it is conducted (e.g. is it engaging community perspectives and creating spaces for reflection on what’s working and what’s not, or is it focused on the kind of bean-counting and report writing that takes staff time away from good implementation?).

Furthermore, how has the dominance of the compliance culture at USAID and other U.S. agencies affected NGOs that receive significant funding from these agencies?  To what extent has the compliance culture also become dominant within NGOs? 

Sherine Jayawickrama manages the Humanitarian & Development NGOs domain of practice – and the Humanitarian & Development NGOs blog – at the Hauser Center for Nonprofit Organizations at Harvard University.

Twaweza, the NGO where I am interning, is an East Africa-wide experiment in using information access and citizen agency to improve public service delivery. Twaweza has an innovative (and largely untested) theory of change. This blog entry explains the theory of change and raises some concerns around the key question: can access to information really bring about change?

The Theory of Change
Twaweza’s theory of change relies on providing ordinary citizens with information in accessible formats about topics that matter to them, particularly education, health, and access to water. Still in its early stages, Twaweza is focusing its first efforts on getting information to ordinary citizens by signing agreements with five networks that can publicize Twaweza’s content. These five networks – mobile phones, media, religious institutions, the teachers’ union, and fast moving consumer goods – reach virtually every person in East Africa.

Twaweza aims not only to reach everyone, but also to provide information in a manner that is accessible and meaningful. An example of this is Uwezo, the Twaweza partner program where I am working. Uwezo will conduct an annual assessment of basic literacy and numeracy skills in students’ homes, reaching about 250,000 children in districts across Kenya, Uganda, and Tanzania this year.

The assessment communicates a key piece of information: primary school enrollments have risen, but many students are not learning. By using a simple, household-based test, the assessment also brings parents into the often opaque world of their children’s education. The overall results have launched national conversations, but many of the most important actions will take place at the village level, as individuals react to their children’s results.

Information and Action
Can information access make a real difference in the quality of public services? Rakesh Rajani, Twaweza’s founder and head, believes so. He intends to use the five networks to establish an ecosystem across East Africa in which citizens can become advocates for themselves. That ecosystem will emerge from an ever-increasing set of feedback loops; ordinary people will receive information through the five networks, will discuss it with each other, will make their opinions heard through radio call in shows, text message reply systems, and more, and will eventually begin to take action.

Daladala TV, a program launched in June by another Twaweza partner, Kilimanjaro Productions, is an example of how Twaweza’s approach will help citizens hear each others’ voices. The show involves sending a local bus (daladala) around Dar es Salaam every morning, outfitted on the inside with a TV studio. On the bus, citizens join in on a talk show about current issues.

Many Tanzanians fear speaking out publicly against the government, so part of the show’s role is to increase citizens’ comfort with public dissent, a key feature of an ecosystem of change. The program is airing daily for six months, featuring the voices of ordinary citizens in prime time.

How will public discussion and dissent lead to action? Rakesh argues that ordinary people are moved to action through comparisons and stories. For example, Daraja, a Twaweza partner organization that works on water issues, may report that in one district 80% of people have access to a functioning water point, while in a neighboring district, only 20% of people do. Citizens in the second district might work through personal connections, such as a successful local business-owner, to pressure the district water engineer to fix their water pumps.

Twaweza would then publicize their story as an example for other citizens of one way they could effect change in their communities. Twaweza hopes that this constant cycle of sharing information and stories will create an ecosystem where citizens successfully advocate on their own behalf.

A Critique of Twaweza’s Approach
The first question this approach raises is the most obvious: will it work? The literature on citizen agency shows mixed results. Some of the more effective experiments have relied on carefully structured citizen pressure, such as using CSOs to create citizen report cards.

Twaweza’s approach, however, depends on people acting on their own or organizing themselves. With a philosophy of avoiding CSOs and NGOs, which are often closely entwined with the power structure, Twaweza intentionally avoids working through pre-existing organizations or helping citizens build formal organizations.

It remains to be seen whether this will work. My hunch is that Twaweza will eventually find that many ordinary people need more active, organized support in order for their actions to have an impact.

A related concern is whether ordinary citizens will know what to advocate for, particularly on complicated issues like education. Having taught in a low-performing school in the U.S., I have seen first hand that even the combination of educational experts, teachers and parents are often unsure how to fix the schools.

If a Ugandan mother learns through the Uwezo assessment that her 11 year old son cannot read, she may want to take action, but how will she know which changes at the school would help her son learn? Although I commend Twaweza for encouraging citizens to act based on knowledge of their own communities, Twaweza may need to offer more concrete suggestions about what actions would be productive.

My last concern is an ethical one. Twaweza’s overall strategy advocates a major cultural shift, from passive acceptance of problems to advocacy and action. Ordinary citizens have adopted this culture of acceptance with good reason; many know people who spoke out against corruption and were arrested on trumped up charges, or whose businesses were closed after they spoke out too loudly.

Should regular citizens act despite their cautious instincts, particularly without a larger organization like Twaweza to defend them from the fallout? Twaweza aims to set events in motion in thousands of villages across the region, but what is its ethical responsibility for the results? Twaweza’s theory of change asks people to take serious personal risks. Although these risks may be necessary for services to improve, Twaweza should continue weigh these ethical questions as it works to build an ecosystem of citizen activism across East Africa.

Twaweza is an exciting and ambitious undertaking. If it can work through the issues raised here, it has the potential to serve as a new model for development, helping ordinary citizens take their rightful place as the drivers of accountability in their own countries.

Ilana Kessler is a Masters in Public Policy candidate at the Harvard Kennedy School and has a 2010 Hauser Summer Fellowship.

Today, as President Obama headed to Canada for the G-8 meeting, the White House released a statement outlining a new U.S. strategy to advance global development. Given the continuing delays – and considerable confusion – that have marked the Quadrennial Diplomacy & Development Review (QDDR) and the whole-of-government development policy review process, it is a relief to hear that President Obama will be issuing a new policy directive on development “in the near future”. 

After months of guessing – and a leaked memo that pretty much spilled the beans – excerpted below is what the White House statement outlines.

President Obama’s new development policy will:

• Foster the Next Generation of Emerging Markets:  The U.S. will intensify efforts to promote sustainable economic development and support good governance by making targeted investments in countries and/or regions where the conditions are right for progress.

• Invest in Game-Changing Innovations:  By leveraging the power of research and development, the U.S. will work to create and scale-up technologies for health, green energy, agriculture, and other development applications.

• Meet Basic Human Needs in a Sustainable Fashion:  The U.S. will continue to be a global leader in the meeting of basic human needs, but will place increasing emphasis on building sustainable public sector capacity to provide basic services over the long-term.

• Tailor Development Strategies:  The U.S. will tailor development strategies in countries in or recovering from conflict to reflect the unique conditions on the ground, and will join efforts to promote stabilization and achieve security with those designed to promote our long-term sustainable development goals.

• Hold all Aid Recipients Accountable:  The U.S. will seek sustained development progress in all countries receiving U.S. economic assistance by placing a greater focus on policy reforms key to development.

In addition, in pursuing these objectives, the U.S. will pursue a new approach to development that:

• Is More Selective:  The U.S. will seek a division of labor with other donors and focus its efforts on select countries, regions, and sectors – while ensuring critical development needs are met.

• Leverages other Donors, Philanthropy, Diaspora and the Private Sector:  The U.S. will seek a division of labor with other donors and make a concerted effort to partner with other actors to leverage U.S. investments.

• Underscores Country Ownership and Mutual Accountability: The U.S. will place a premium on partnering with countries that are well governed and will work to strengthen their institutions and support their development strategies.

• Strengthens Multilateral Capabilities:  The U.S. will support multilateral development capabilities and support key reforms and the creation of new capabilities, where required.

• Drives Policy with Analysis:  The U.S. will adopt metrics and set in place rigorous standards for monitoring and evaluation, and use data and analysis to drive decision-making.

Many of these are welcome directions. The emphasis on country ownership, evidence-based decision-making, multilateral capabilities and focused investments is strategic. What is not crystal clear is what President Obama articulates as the central purpose of advancing development. 

Given what we see in the U.S. National Security Strategy, it’s possible to surmise that development is advanced because it is important for U.S. national security. Yet, President Obama and several senior administration officials clearly believe that development is important in its own right.

Many argue that this is a false dichotomy but, unless the purpose question is answered with clarity, confusion and fragmentation will continue to reign even after the directives and reviews are finalized.

Sherine Jayawickrama manages the Humanitarian & Development NGOs domain of practice (and this blog) at the Hauser Center for Nonprofit Organizations at Harvard University.