Monitoring and Evaluating NGO Achievements

MandE NEWS > Documents > Monitoring and Evaluating NGO Achievements

Monitoring and Evaluating NGO Achievements

By Rick Davies, Cambridge, UK.
rick@mande.co.uk, www.mande.co.uk

To be included in the
Arnold Companion to Development Studies,
edited by Potter, R.B., and Desaid, V., to be published by Arnold in early 2001

DIFFERENT PERSPECTIVES

Over the last decade there has been a dramatic growth in the number of NGOs involved in development aid, in both developed and developing countries. The total amount of public funds being spent through NGOs has grown dramatically and the proportion of development aid going through NGOs, relative to bilateral or multilateral agencies, has also increased.

Associated with this growth has been a growing concern about identifying the achievements of NGOs. This has been evident in the burgeoning literature on the monitoring and evaluation of NGO activities. There has been a steady stream of experimentation with specific methods, especially those focusing on participatory approaches to M&E and impact assessment (e.g. IIRR, 1997; Goyder et al, 1997; Abbott and Guijt, 1998; Guijt, 1998). On a smaller scale, a number of NGOs have produced their own guides on monitoring and evaluation (Gosling and Edwards, 1995; Platt, 1996; Broughton & Hampshire, 1997; Barton, 1997; Rubin, 1998; Roche, 1999). Recent books on NGO management are giving specific attention to assessing performance (Fowler, 1997) and the management of information (Powell, 1999). As well as doing their own evaluations, some NGOs are now doing meta-evaluations (of methods) and syntheses (of results) of their evaluations to date (Mansfield, 1996; Evison, 1999, Plan International, 2000). Similar but larger scale studies have been commissioned by bilateral funding agencies (Riddell et al, 1997; Oakley, 1999; AusAid, 2000). Both sets of studies have attempted to develop a wider perspective on NGO effectiveness, looking beyond individual projects, across sectors and country programmes. They have been more critical and analytic, when compared to the more prescriptive and normative approach of the method literature. Overall, NGOs have become much more aware of the need for evaluation, compared to the 1980s when there was some outright hostility (Howes, 1992: 393).

In contrast to this operational literature, there are relatively few examples of a more independent literature, which steps back and looks at the institutional context in which monitoring and evaluation activities are taking place (e.g. Carlsson, et al, 1994). So far the main focus has been on bilateral and multilateral donor influences, within a wider perspective than monitoring and evaluation alone (Wallace, et al., 1997; Edwards and Hulme, 1996). Less evident are more economic perspectives, looking at NGOs as agents in a marketplace (Maren, 1995; Sogge, 1996), although most large NGOs have marketing departments and these pre-date and exceed in size other units dealing with government funding. Further afield, but growing in number, are the very public criticism of NGOs as a whole, which contain implicit and explicit judgements about NGO performance (de Waal, 1998; Maren, 1998; Shawcross, 2000; Bond, 2000). At the other extreme of involvement are the more biographical accounts of NGO work, which include insiders perspective on monitoring and evaluation practices (Morris, 1991). One small but emerging genre is the organisational ethnography (Harper, 1998; Crewe and Harrison, 1999). These have been widely used in research into the application of information technology, and more generally in the sociology of science (Anderson, 1994).

WHAT DO THEY KNOW?

The DAC (Riddell et al, 1997) study "Searching for Impact and Methods: NGO Evaluation Synthesis Study" is the most comprehensive overview of NGO impact, and impact evaluation methods to date. This study looked at evidence from 60 separate reports of 240 projects undertaken in 26 developing countries. On the first page of the executive summary the authors report that:

"A first, overarching, conclusion - confirmed by data and interviews in all the different case study countries - is that in spite of growing interest in evaluation, there is still a lack of reliable evidence on the impact of NGO development projects and programmes."

In their conclusions about enhancing impact in the future they note that:

"…a repeated and consistent conclusion drawn across countries and in relation to all clusters of studies is that the data are exceptionally poor. There is a paucity of data and information from which to draw firm conclusions about the impact of projects, about efficiency and effectiveness, about sustainability, the gender and environmental impact of projects and their contribution to strengthening democratic forces, institutions and organisations and building civil society. There is even less firm data with which to assess the impact of NGO development interventions beyond discrete projects, not least those involved in building and strengthening institutional capacity, a form of development intervention whose incidence and popularity have grown rapidly in the last five years." (Riddell, et al., 1997: 99).

Similar conclusions were reached by the recent Danida-funded study of 45 Danish NGO projects in four countries (Oakley, 1999: 94)

These two multi-country studies raise serious doubts as to whether many NGOs know what they are doing, in the sense of their overall impact on people's lives. NGOs may or may not be having a positive impact, but their ability to scale up that impact must be limited by the ability to evidence those achievements (and their limitations) and communicate this information to others with more resources and / or influence.

The significance of this lack of knowledge needs to be kept in perspective, worrying as it is. Thin (1999a: 12) notes that in 1999 only 6% of DAC (bilateral agency) evaluation abstracts make any reference to poverty. This is despite the fact that all major bilateral agencies now see poverty reduction as their primary objective.

WHAT IS THE PROBLEM?

Given the hundreds, if not thousands, of millions of pounds that have been spent by NGOs over the last decade why has it been so difficult to come to persuasive conclusions about the results of their work? Several different reasons are examined below, including those proposed by Fowler (1997; 160) and Riddell et al. (1997; ix).

Ambitious expectations

In the DAC review Riddell et al. (1997: 66) noted that almost all "the Terms of Reference [for evaluations] set the scene for anticipating exceedingly high expectations of what can be achieved, particularly what can be said about development impact. In quite sharp contrast, the tone of the conclusions is usually cautious and tentative, arguing that it is difficult to come to firm and decisive conclusions…". Both the DAC and Danish NGO studies used nine different performance criteria to compare NGO projects. The proposed SPHERE (2000) Training Module on Monitoring and Accountability lists 10 different criteria. Most of these are in addition to what are often a quite ambitious set of objectives defined within a project's Logical Framework(2). However, unlike the contents of these Logical Frameworks there must be some doubt as to whether many NGOs knowingly sign up to all of these additional and ambitious expectations at the time when they seek approval and funding for the project.

Complexity caused by scale

Expectations of project performance are raised even further by the hierarchical structure of large NGOs, and their position in a larger hierarchy of associated partner organisations (including both implementing partners and back-donors). Large international NGOs can have country, regional and international strategies. At each level their strategy documents will list a number of objectives. Donors, such as DFID, will in turn have their own international objectives and targets. This plethora of objectives is only manageable if objectives are clearly nested, such that local objectives are detailed versions of more macro-level objectives. In these circumstances judgements about the smallest units can be used as raw material for judgements of larger units that they belong to (Fowler, 1997: 169). In practice, doing so is complicated by the heterarchical nature of aid supply chains. Individual NGOs often belong, through their donors, to more than one hierarchy of organisations, each with differing expectations and reporting practices. In the UK, the largest NGOs are still struggling with the complex issues associated with aggregating their experience on a large scale. (Davies, 2000).

Diversity of Activities

The majority of the largest NGOs tend to be generalists, being involved a wide range of development activities, across a number of sectors (Oakley, 1999). This must make the task of comparing and aggregating performance information more difficult. The problems of diversity are accentuated by progressive growth in organisational scale. One response to diversity of practice within many NGOs has been to initiate more thematic studies, which focus on one type of activity (e.g. micro-finance, health, water, etc.), but across a number of countries. Another response has been to develop assessment methodologies tailored to specific types of interventions (e.g. MEASURE and SPHERE). Both responses manage diversity through specialisation. A less common response has been to reduce the scale of the task, but to maintain a more holistic focus, by undertaking within-country cross-activity studies. Each option carries with it an implied judgement about the type of knowledge that matters (sectoral versus country based), and whose judgement matters. In contrast to single-sector studies, country-based studies offer more potential to build in public participation and accountability, at each level of aggregation.

Vague objectives

It is widely recognised that the achievements of many development objectives, such as empowerment, institutional strengthening and the development of civil society, are difficult to define in advance. Evidence of their achievement is not easy to agree on, and there is no one single path to their realisation. The value of what is achieved often depends local context and history. Establishing pre-defined near-universal indicators for such changes is inherently difficult. This is a problem if measurability is over emphasised, as it seems to be in the case of many evaluations (Thin, 1999a: 26). However, many important development events are not measurable, but they are verifiable. News stories, found in reputable newspapers, are a good means of reporting exceptional and significant events (Davies, 1998; Dart, 1999), so much so that people are prepared to pay for them. On the other hand, they are less suitable as the basis for generalisations. The significance of this problem depends on the scale at which totalising statements are required, and this in turn, on the overall structure of the aid sector.

The tools being used.

Fowler has argued that the "limitations of the instruments that NGOs use to monitor, evaluate and review" (1997: 160) are one reason why NGOs have not been able to substantiate their achievements. Certainly there has been a lot of argument over the value of the Logical Framework as a planning and monitoring tool (Wallace, et al, 1997). Logical Frameworks have been useful in encouraging the identification of indicators at the planning stage, but much less so in ensuring their actual use during project monitoring or evaluation (Davies, 1997). In practice, the widespread focus on identification of indicators reflects a bias towards planning rather than monitoring and evaluating that is built into most NGOs, and other agencies. In the worst case the advocates of Logical Frameworks have promoted a very narrow view of indicators i.e. only that which is measurable can be managed (Team Technologies Inc, 1994).

Outside of the Logical Framework many NGOs have been actively searching for appropriate methods, especially in ways of enabling people' participation in the monitoring and evaluation of projects (IIRR, 1997; Guijt, 1998). The challenge here has been how to aggregate the complex and large volume of information and analysis generated by these methods (Goyder et al, 1997).

Methodological developments have been less noticeable with analysis of performance above field and project level. At the country programme level and higher, Logical Frameworks have been pre-empted by strategic planning frameworks, suggesting that Logical Frameworks are not scalable solutions to planning and monitoring. With some applications of strategic planning NGO s have found they need to assess their country programmes according to multiple strategic objectives, rather than one goal level statement (Oxfam, ActionAid). While this plurality allows some variation in strategic emphasis between different countries it does complicate the task of inter-country comparisons and coming to high level generalisations about achievements.

The absence of baseline information and adequate monitoring systems

The absence of adequate baseline information, is an almost universal complaint found in both NGO and donor meta-evaluations / synthesis studies (Mansfield, 1996; Riddell et al, 1997; Oakley et al, 1998; Evison, 1999) Another less noted phenomena is the incidence of base-line survey data being lost or forgotten, and unavailable to evaluation teams (Goyder, 1997). Although most organisations have monitoring systems of some sort, many writers (Fowler, 1997: 169; Riddell, 1997; Roche, 1999) have noted the pervasive problem of organisations monitoring expenditure, activities and outputs, but not effects and impacts. All of these phenomena are really symptoms rather than explanations of why NGOs do not seem to know what impact their work is having. If some types of information are not being produced then we need to ask why is there no demand for that information.

Organisational structure and relationships may be a more significant factor than the absence of appropriate M&E concepts or methods. Inside most organisations proximate rationality rules. Activities are measured against activity plans, expenditure against budget. These are immediate tasks where delays are visible and have consequences for those responsible. Staff have to cope with the short term before they can worry about the long term. On the other hand there are external demands for information about performance, arising primarily from donors and governments. Financial reporting is required most often, then implementation relative to plans, then much less frequently, achievement in terms of changes in peoples lives (purpose and goal level type statements). Not being dependent on their clients for their financial survival, NGOs' incentives to attend to clients' judgements about effects and impact are dependent on organisational culture and values.

Variations do exist between NGOs in terms of their dependence on bilateral and multilateral donors versus individual members of the public. The nature of this dependence is likely to effect the information demands being made on those NGOs, and their motivation and capacity to assess their achievements. Individual members of the public donate to charities on the basis of trust, and that relationship is managed by marketing departments, not M&E units. There is some anecdotal evidence to suggest that the NGOs with the most financial independence from official donors have invested the least in monitoring and evaluation (e.g. World Vision versus CARE International). Similar contrasts can be found amongst UK agencies.

Amongst NGOs with significant funding from bilateral agencies the picture is more mixed. Wallace et al (1997) have argued that donors such as DFID are responsible for the widespread use of the Logical Framework, by British NGOs. On the other hand, almost all the Danish NGOs in the INTRAC study have received the majority of their funding from Danida, but without any significant impact on their capacity to monitor and evaluate their work (Oakley, 1999).

WHERE TO NEXT?

Many of the problems discussed above relate to the scale and structural features of individual organisations, and their place in the wider ecology of aid organisations. The structure of information flows between those organisations is now open to the possibility of radical change, because of the increasing accessibility of the internet

Most of the large bilateral aid agencies are now making their evaluation reports publicly accessible on a global scale, via the World Wide Web (See www.mande.co.uk/sources.htm ). A small number of NGOs have done the same. The next step forward in transparency, and more immediate public accountability, would be for those organisations to place their annual progress reports in the public domain as well, via the WWW. Another step forward, already take by organisations such as Christian Aid, is to provide hypertext links to their own southern partner NGO web sites, allowing outsiders more direct access to documented accounts of aid funded activities written by those closer to the action. In effect the internet is opening up the possibility of devolving progress reporting to the public (northern and southern), to organisations much further down the aid supply chain. Associated with this is the possibility of those organisations gaining more direct and global access to public funds. This dis-intermediation process, already seen in the private sector, may require northern NGOs to rethink how they add value in the aid supply chain. One option will be for Northern NGOs to focus on building partners' capacity to report this way, and on monitoring public interaction and reaction to the information provided. Development education and public accountability concerns could be addressed at the same time, while also providing greater opportunities for lateral learning between NGOs. Value could be unlocked from today's NGO "conglomerates".

A GUIDE TO FURTHER READING

Monitoring and Evaluation NEWS at www.mande.co.uk

This web site includes details of a large amount of grey literature not yet available in journal and books, including email addresses and / or web pages where the full texts can be obtained. The site also includes information on other related web sites, coming events (including training, workshops and meetings), an editorial section, book reviews, and lists of vacancies for M&E specialists. The site is funded by six British NGOs.

Riddell, R. C. , Kruse, S-E., Kyollen, T., Ojanpera, S., Vielajus, J-L. (1997) Searching for Impact and Methods: NGO Evaluation Synthesis Study. A Report produced for the OECD/DAC Expert Group on Evaluation. Department for International Development Cooperation, Ministry of Foreign Affairs. Helsinki. http://www.valt.helsinki.fi/ids/ngo.

This is the most definitive study of NGO impact to date, primarily because of the breadth of its coverage, and detailed analysis. The primary purpose of the study was to undertake a synthesis study of the impact of Non-Governmental Organisation (NGO) development projects, derived largely from evaluation reports, as well as the methods used in assessing impact. The information was gathered from evaluation reports commissioned by donors, and from data and information gathered (through reports and interviews) in 13 country case-studies, undertaken in both donor and southern countries. The overall Study comprises two volumes, The Main Report (125 pages), and The Appendices (375 pages) which contain the case studies and an extended bibliography.

Roche, C. (1999) "Impact Assessment for Development Agencies: Learning to Value Change". Oxfam. Oxford.

Based on original research undertaken by Oxfam GB, Novib and a number of agencies around the world, this book draws on in-depth case studies undertaken in a range of settings, from large scale integrated development programmes to small-scale community development in initiatives. The book gives a brief overview of the theory and discusses the different approaches to the design and implementation of impact assessment processes. The main part of the book explores the different tools and methods used in the case studies and illustrates their uses in emergency work, in advocacy, and in the analysis of social relations and of organisations. The final chapters look at best practice in the development of impact assessment processes and systems in organisations as well as some of the policy lessons that emerge from the findings of the studies., and based on a review of past experience.

Thin, N. (1999a) Methods and Approaches for Evaluation of Development Assistance for Poverty Reduction: A Literature Review. University of Edinburgh, Edinburgh.

This very comprehensive review was prepared as a background paper for OECD-DAC Working Party on Aid Evaluation Workshop on Evaluation of Poverty Reduction, Edinburgh 12-14 October 1999 . Subject headings include: Poverty and Poverty Reduction; Evaluation and Knowledge Management; Evaluation: Methods, Tools, and Dimensions of Information; Suggestions for Further Actions and Investigation.

Thin, N. (1999b) Evaluation of Poverty Reduction: Thematic Review And Annotated Bibliography. University of Edinburgh, Edinburgh.

This is a sister volume to the above text. More than 150 pages in length it is very comprehensive. Subject headings include: Poverty analysis: definitions, causes, objectives and indicators; Poverty reduction policies and strategies, Evaluation Methods, approaches and vehicles; Evaluation in diverse sectors and kinds of interventions; Websites and databases.; Ongoing and future initiatives in EPR.

References:

Abbot, J., Guijt, I. (1998). Changing views on change: participatory approaches to monitoring the environment. SARL Discussion Paper No. 2, July 1998. pp96. ISBN 1560-2192 London. IIED

Anderson, R.J. (1994). "Representations and requirements: The value of ethnography in system design. " Human-Computer Interaction 9: 151-182.

AusAid (2000) Personal communication with Irene Wettenhall,

Performance Information and Assessment Section, AusAID, Canberra

Barton, T. (1997) Guidelines to Monitoring and Evaluation: How Are We Doing? CARE Uganda.

Bond, M.C. (2000) Backlash Against NGOs. Prospect, April. London

Broughton, B., Hampshire, J. (1997) Bridging The Gap: A Guide To Monitoring And Evaluating Development Project. Australian Council for Overseas Aid. Canberra.

Carlsson, J., Kohlin, G., and Ekbom, A. (1994) The Political Economy of Evaluation: International Aid Agencies and the Effectiveness of Aid. St Martin's Press. London.

Crewe, E., and Harrison, E. (1999) Whose Development? An Ethnography of Aid. Zed Books, London.

Dart, J. (1999) Target 10 Evaluation Stories May 1998 - May 1999. Narrative Approach For Monitoring The Non-Economic Impact Of The Target 10 Project. Melbourne.

Davies, R.J. (1997) Donor Information Demands and NGO Institutional Development. Journal of International Development. Vol. 9. No. 4. Pp613-620.

Davies, R.J. (1998) An Evolutionary Approach To Facilitating Organisational Learning: An Experiment By The Christian Commission For Development In Bangladesh. Impact Assessment And Project Appraisal. March Vol. 1., Number 1. p243-250.

Davies, R.J. (2000) Interviews with M&E sections of Oxfam, SCF, ActionAid, Plan, and World Vision, in February.

de Waal, A. (1998) Famine Crimes : Politics & the Disaster Relief Industry in Africa Indiana Univ. Press.

Edwards, M., Hulme, D. (1996) Too Close for Comfort? NGOs, States and Donors. Earthscan. London.

Evison, I. (1999) Learning from Evaluations in Christian Aid: A Critical Analysis Of Evaluation Reports 1991-98. Christian Aid. London.

Fowler, A. (1997) Striking A Balance: A Guide To Enhancing The Effectiveness Of Non-Governmental Organisations In International Development. Earthscan. London.

Gosling, L., Edwards, M. (1995) Toolkits: A Practical guide to Assessment, Monitoring, Review and Evaluation. SCF. London.

Goyder, H., Davies, R., Williamson, W. (1997) Participatory Impact Assessment: A Report On A DFID Funded Actionaid Research Project On Methods And Indicators For Measuring The Impact Of Poverty Reduction. ActionAid. London. See also http://www.swan.ac.uk/cds/guidance.htm.

Guijt, I. (1998) Participatory monitoring and impact assessment of sustainable agriculture initiatives. SARL Discussion Paper No. 1, July 1998. pp112. ISBN 1560-2192 London. IIED.

Hancock, G. (1992) Lords of Poverty : The Power, Prestige, and Corruption of the International Aid Business. Atlantic Monthly Press

Harper, R.H.R. (1998) Inside The IMF: An Ethnography Of Documents, Technology And Organisational Action. Academic Press. San Diego. See www.amazon.co.uk for description and synopsis.

Howes, M. (1992) Linking Paradigms And Practise. Key Issues In The Appraisal, Monitoring And Evaluation Of British NGO Projects. Journal of International Development. Vol. 4, No. 4.

IIRR (1997) Participatory Monitoring and Evaluation: Experiences and Lessons. Workshop Proceedings 24-29 November 1997. International Institute of Rural Reconstruction. Cavite.

Kershaw, I (1995) Review of the Effectiveness of NGO Programs. AusAid. Canberra.

Law, J. (1994). Organizing Modernity. Oxford, Blackwell.

Monitoring and Evaluation NEWS web site, at www.mande.co.uk

Mansfield, D. (1996) Evaluation Tried And Tested? A Review of SCF Evaluation Reports. SCF Working Paper No. 17. Save the Children Fund. London.

Maren, M. (1995) A Different Kind of Child Abuse. Penthouse Magazine. December 1995.

Maren, M. (1998) The Road to Hell : The Ravaging Effects of Foreign Aid International Charity. Free Press. New York

Morris, T. (1991) The Despairing Developer: Diary of an Aid Worker in the Middle East. London. Taurus.

Oakley, P., Pratt, B., Clayton, A. (1998) Outcomes and Impact: Evaluating Change in Social Development. INTRAC. Oxford.

Oakley, P. (1999) Overview report. The Danish NGO Impact Study. A Review of Danish NGO Activities in Developing Countries. INTRAC. Oxford. Full text is also available at: www.um.dk/udenrigspolitik/udviklingspolitik/ngosamarbejde/impact/syntese_eng/

Plan International (2000) Personal Communication with Sam McPherson, Evaluation Officer, London. March 2000.

Platt, I (1996) A Review of Participatory Monitoring and Evaluation. Concern Worldwide. Dublin.

Powell, M. (1999) Information Management for Development Organisations. Oxfam. Oxford.

Riddell, R. (1992) Judging Success: Evaluating NGO Approaches To Alleviating Poverty In Developing Countries. ODI Working Paper 37. ODO. London.

Riddell, R.C., Kruse, S-E., Kyollen, T., Ojanpera, S., Vielajus, J-L. (1997) Searching for Impact and Methods: NGO Evaluation Synthesis Study. A Report produced for the OECD/DAC Expert group on Evaluation. Department for International Development Cooperation, Ministry of Foreign Affairs. Helsinki. http://www.valt.helsinki.fi/ids/ngo.

Roche, C. (1999) "Impact Assessment for Development Agencies: Learning to Value Change". Oxfam. Oxford.

Rubin, F. (1998) A Basic Guide to Evaluation for Development Workers. Oxfam. Oxford.

Shawcross, W. (2000) Deliver Us from Evil : Peacekeepers, Warlords and a World of Endless Conflict. Simon & Schuster. New York.

Sogge, D. (Ed) (1996) Compassion and calculation: The Business of Private Foreign Aid. Pluto Press. London.

SPHERE (2000) SPHERE and Accountability - Monitoring and Evaluation. The SPHERE Training Programme. Module 4 - One Day. Draft document.

Team Technologies Inc. (1994) Teams and Project Design. TeamUP Workbook.

Thin, N. (1999a) Methods and Approaches for Evaluation of Development Assistance for Poverty Reduction: A Literature Review. Draft background paper for OECD-DAC Working Party on Aid Evaluation. University of Edinburgh. Edinburgh.

Thin, N. (1999b) Evaluation of Poverty Reduction: Thematic Review And Annotated Bibliography. University of Edinburgh, Edinburgh.

Tvedt, T. (1998) Angels of Mercy or Development Diplomats : Ngos & Foreign Aid; Africa World Press Inc.

Wallace, T., Crowther, S., Shepherd, A. (1997) Standardising Development - Influences On UK Ngos' Policies And Procedures. World View Publishing. Oxford

12 April 2000

Endnotes

(1) A summary representation of a project in table form, describing project activities, outputs, purpose and goal in four rows, and a narrative summary, measurable indicators, means of verification and assumptions for each row, listed in four columns.

Return to Rick Davies - Home Page or MandE NEWS