Return to MandE NEWS

Methods and Approaches for Evaluation of Development Assistance for Poverty Reduction

A Literature Review

Neil Thin

University of Edinburgh

with assistance from Beth Mellor

Draft background paper for OECD-DAC Working Party on Aid Evaluation

Workshop on Evaluation of Poverty Reduction, Edinburgh 12-14 October 1999

Executive summary

1. The paper proposes a framework for structuring the analysis of evaluation of poverty reduction (EPR), following a logical sequence through identification of problems that are defined under the ‘poverty’ rubric and their causes and dynamics, through the policies, strategies, and instruments, to the evaluation process itself (systems, approaches, and study designs, and the methods and tools which will best serve those evaluation systems). (1.1)

2. Key relevant changes in aid management are summarised. These include: the almost universal assertion of poverty reduction as the fundamental objective of aid, together with adoption of International Development Targets (IDTs); increased attention of aid agencies to broader influences on poverty such as trade, economic management, and civil/political stability, and consequently greater emphasis on learning, innovation, and knowledge sharing; and increased promotion of multidimensional concepts of poverty and well-being, rights-based approaches, gender equality, and empowerment of poor people (1.2.1)

3. These changes in aid approaches are linked with new emphases in approaches to evaluation and knowledge management. Evaluation work is being upscaled from project to sectoral and country levels. Several donors are taking an interest in Results-Based Management. There is broad consensus on the importance of balancing quantification with qualitative analysis and interpretive presentation of information on poverty. There is increasing emphasis on evaluation as a set of long-term planned processes of learning, adaptation, and knowledge management rather than as discrete one-off snapshot studies - and linked with this a much greater emphasis on strengthening partners’ capacity for knowledge management. (2.1.2)

4. The paper reviews various approaches to the conceptualisation of poverty and links these with classification and analysis of poverty reduction strategies as an essential pre-requisite to planning and reviewing EPR. Generally, the evaluation of processes is becoming increasingly emphasised due to increased attention to dynamics of poverty rather than just poverty ‘snapshots’. But analysis of poverty and poverty reduction is still hampered by weak conceptualisation. Examples of approaches to distinguishing poverty reduction strategies are discussed. Drawing on the classification systems found in donor and academic literature, the paper argues for a more analytical rather than ‘labelling’ approach which evaluators might use to sharpen up their identification of various paths from aid interventions to poverty reduction. To facilitate this, a ‘SPOT check’ is recommended as a minimal set of four axes of distinction (2.2):

   Strategic distance (direct/indirect approaches)

   Poverty dynamics (relation of an intervention to cause-effect patterns)

   Operational level (international, macro, meso, or micro)

   Targeting (intended direct/indirect beneficiaries)

5. Such analysis reveals that overwhelmingly, donor literature which is explicitly on ‘poverty reduction’ (policy, guidance, input analysis, and evaluation literature) has tended to focus on relatively direct approaches to poverty reduction, on work which addresses either symptoms or proximate causes of poverty, on micro-level operations, and on targeted interventions. It seems likely that this focus is driven the institutional imperative to identify aid-attributable poverty-reducing outcomes - i.e. those with the simplest and most visible cause-effect linkages. The implication is that if evaluation departments want to learn something new about how poverty reduction is achieved, they should be investing more effort into assessing the more complex and less well-understood cause-effect linkages such as outcomes from international-level interventions (such as debt relief, human rights advocacy, development awareness-raising, and work designed to control international migration and drug-smuggling), and the roles of meso-level institutional strengthening in linking macro-level policies to micro-level realities. (2.2.9)

6. Partly consequent on the above weaknesses in classification, the labelling of different kinds of poverty focus in evaluation work is far too inadequate to facilitate synthesis of lessons across different kinds of intervention and agency. Despite the centrality of poverty in donor mission and goal statements, in most evaluation studies poverty issues tend still to remain at best implicitly dealt with, and certainly not highlighted in evaluation summaries. Even in the 6% of DAC summaries that mention poverty, few have any substantial comment to make about poverty reduction. Interestingly the converse is also true: comparatively few policy and analytical papers on poverty have anything explicit to say about evaluation. (2.4)

7. The question is raised of whether ‘poverty reduction’ is a distinctive sub-category of aid evaluation work. This review finds that it is not, given a) that almost all donors now see poverty reduction as their over-arching objective, and b) the danger that the ‘EPR’ rubric might distract attention from the less obvious connections between aid and poverty reduction. There will be a greater chance of showing the poverty-relevance of all or most aid evaluation work if more specific sub-categories are devised and applied, differentiating among approaches to poverty reduction and operational levels. Promoters of poverty focus, who want to avoid the ‘direct poverty alleviation’ trap, could do well to learn from the experience of upstreaming gender concerns (avoiding the ‘WID trap’ which similarly marginalised the issue by focusing on immediate practical needs). Like gender mainstreamers, poverty mainstreamers need to ensure that causes of inequality are addressed at all levels of operation and influence.(2.4)

8. The inherent optimism of the term EPR is potentially problematic unless strenuous efforts are made to ensure that evaluators cover cases where poverty is increasing, including those where poverty increase is in part attributable to aid interventions. The review has found little disappointingly little sign of serious rigorous analysis of counterfactual scenarios linking attribution of project impacts with hypothetical without-project changes in the poverty situation (for better or worse).Most evaluation work is also still overwhelmingly focused on achievement of objectives rather than unintended outcomes, with the result that aid’s adverse impacts on poverty go under-recorded. (2.5)

9. EPR could be helped by broader emphasis on fungibility: when evaluators look above project levels to assess net outcomes, they must a) address fungibility by looking beyond the withdrawal of aid, and b) their analysis of fungibility must go beyond budgetary and governmental fungibility to include all significant ways in which aid-supported effort displaces other efforts. (3.3)

10. Evaluation work affects incentives for various kinds of development work, and does not necessarily steer effort towards the most effective poverty reduction work and the neediest people. There are some signs that evaluators and senior managers are under-emphasing external factors, over-emphasing statistical ratings on performance, and mistakenly expecting implementers to be held ‘accountable’ for their achievement of impact on poverty. As poverty reduction and the assessment of results become more prominent in aid agency priorities, evaluators are likely to come under pressure to produce findings which attribute credit for poverty reduction to individuals, projects, and agencies such that they can be rewarded for good performance. It will be hard to achieve this without distorting efforts towards directly attributable and tangible results, low-risk interventions, and single-agency flag-waving. (3.4)

11. The paper discusses the rapid rise in donor interest in strengthening the capacities of partner agencies for evaluation. It is not yet clear whether such efforts are steering evaluators’ attention towards poverty reduction. Also, most such efforts seem to have focused on national governments, and there have been fewer comparable efforts to look at evaluation and knowledge management capacities of civil society and local government organisations (3.7)

12. The paper discusses and examplies the adaptation of evaluation methods to different contexts and strategies, but makes no attempt at comprehensive coverage. It is likely that a separate product from this review will be an annotated bibliography covering some of the immense literature on this subject. (4.1)

13. Multidimensional concepts of poverty demand greater emphasis on qualitative and interpretive dimensions of assessment. Debates about complementarities and trade-offs between qualitative interpretation and quantification are addressed here by emphasising that all quantitive information is dependent on qualitative analysis and judgement, by viewing quantification of poverty reduction primarily as a rhetorical device, by outlining both its benefits and its dangers, and by emphasising the diffuse rather than specific influences of well-presented qualitative information. (4.2, 4.3)

14. There are some signs that ‘evaluability’ (and particularly ‘measurability’) may be being used consciously or otherwise as a criterion for defining strategies and priorities for implementation. Poverty reduction efforts should be guided by their judgement about what is good to do, not what is easy to evaluate; and evaluability should not be confused with measurability (4.3.3)

15. Bearing in mind the powerful rhetorical force of numbers, evaluators may need to put more effort into enhancing the rhetorical force of qualitative and interpretive information. (4.3.6)

16. Participatory learning methods have justifiably grown in importance, but still have three kinds of weakness: they tend to relate only to the micro-level of grass-roots projects, so need to be scaled up; they often fail to spell out operational recommendations for application in poverty reduction strategies; and they are typically lacking in aggregative and comparable information (4.4.6)

17. The same arguments that evaluation units have been using to promote greater donor efforts in support of national evaluation capacity-building are just as valid at meso- and micro-levels. (4.4.4)

18. Various suggestions are made for further actions and investigation, concerning:

    · the process of defining the different ways in which all aid evaluation can improve understanding of poverty reduction

    · further research and development of approaches to the evaluation of very indirect and diffuse international and macro-level instruments for poverty reduction

    · ensuring that outputs from qualitative, interpretive, and participatory approaches to EPR is disseminated in usable and influential forms

    · modifying the DAC evaluation database could be modified so as to facilitate learning about poverty; greater emphasis on unintended impacts on poverty - both desirable and undesirable

    · more development of indicators and assessment of social development, with stronger emphasis on qualitative interpretation, on rights and empowerment, and on inequality at all levels from intra-household inequality to international inequality.

Return to MandE NEWS