Best Evaluation Practice in '97

Best Evaluation Practice in 1997
Edited by Rick Davies, and last updated 16th April 1998.
MandE NEWS is supported by: Oxfam - (UK and Ireland), Save the Children Fund (UK), Intermediate Technology Development Group (UK), and ActionAid (UK).

The Background:

From mid-February 1998 MandE NEWS has extended an invitation to a range of NGOs to provide their own answers to the following question: "In your judgement, which of all the evaluations that you carried out (or funded) in 1997 was the best ?" We want a brief description of what was evaluated, and an explanation of why the evaluation process or method was thought to be especially good. We also want a contact email address where readers can go to for further information about that evaluation.

Go to responses by: VSO...ITDG...Oxfam UK&I...Water Aid...ActionAid....CAA Australia, and DfID, UK

Responses have also been sought from CAFOD, and SCF (30/3/98). Submissions from any other development agency would also be welcome.

Readers responses to any of these submissions would also be welcomed

VSO: "I think our contribution would be our Review of the VSO Namibia Programme, 1997, by Nicky May. This was our most recent review undertaken of a new country programme, some 5 years into its life. It set out to document the history and development of the programme, with rationale, and the process of the review very deliberately supported the work of planning and developing the programme for the next period of time. The work of the external reviewer was complementary to the process through which programme staff had been developing their own analysis in conjunction with local external input. Whilst not particularly innovatory in terms of methodology, this review encouraged and valued input from a variety of stakeholders and effectively served its purpose. The person to contact would be me(Jo Rowlands) in the first instance. If appropriate I could refer people to either the author or the person who was programme director at the time of the review.".

Go to responses by: VSO...ITDG...Oxfam UK&I...Water Aid...ActionAid....CAA Australia, and DfID, UK

ITDG: "Rainwater Harvesting in Turkana: An Evaluation of Impact and Sustainability (Watson and Ndung'u, 1997). This is ITDG's first serious attempt to revisit a past project to look at long term impact and sustainability. It concludes that the project (which finished about five years ago) has had a significant if modest impact. More importantly, it shows that the participatory approaches adopted, which were relatively new in the early 1980s, have led to a high degree of sustainability. The exercise was relatively cheap and methodologically simple. It demonstrates how simple but useful quantitative data can be derived from group and individual discussions. The final report contains no startling insights but is 'best practice' in that it takes long term capacity building and sustainability seriously and should inform other, ongoing work both as regards project implementation and evaluation methodology. Regards Alex Bush, ITDG (alexb@itdg.org.uk)"

Go to responses by: VSO...ITDG...Oxfam UK&I...Water Aid...ActionAid....CAA Australia, and DfID, UK

OXFAM: "Beginning as we mean to go on: Best practice in M&E in Wajir, Kenya.. Wajir is the third largest district in Kenya and has suffered from cyclical drought, aggravated by conflict. It is estimated that at least 70% of the population are nomadic pastoralists. The Oxfam Wajir Pastoralist Development Project (WPDP), established in 1994, is reducing the poverty and vulnerability of the pastoralist and settled communities in this District. Why is it effective?

Building on Oxfam+s previous long-term engagement in the area, WPDP has been planned in a fully participatory manner since it began.
Seeing it as an investment priority, Oxfam sent a three-person team from a local voluntary group it had worked with before, to carry out extensive PRA work to ensure things were got right, from the start, including monitoring and evaluation.
The project is expected to benefit over 100,000 people in its 9 year lifespan.
The project has built up representative structures through the development of 5 Pastoralist Associations.

What is the assessment? Mid-term evaluations see the project as very successful. Important ingredients have been:

The monitoring and evaluation process. The project planned for the collection of germane baseline data from the start which enabled comparative assessments later.
Evaluation has been participatory from the beginning. It has made use of PRA methods and has consulted widely amongst stakeholders.
M and E has been constant and processual, feeding back in a timely and effective manner into planning as the project progressed. For example, the approach for training pastoralist associations was changed after M & E work showed the previous approach was ineffective.
Two people have been employed in M and E throughout the project. This has resulted in excellent project documentation and a huge amount of data collected. For an under resourced, marginalised district like Wajir, this information just did not exist before. It has been used not only in assessments and planning within Oxfam and but also at district government level.
One weakness has been analysis of raw data once collected, which has not been as strong as the programme and situation analysis. Since on-going M & E also highlighted this weakness, the project has been able to respond by sending a M & E officer for training in this area.
A further strength is that the on-going M & E has been combined with external evaluations so that fresh ideas and perspectives are incorporated.

Further documentation is available: Phase 1 Proposal, September 1994; Wajir Mid-Term Review, March 1996; Phase 2 Proposal, April 1997; Building Pastoral Associations in Wajir: Oxfam's Experience, April 1997; Economic Impact Assesssment, January 1998."

For further information, E-mail Izzy Birch atibirch@oxfam.org.uk

Go to responses by: VSO...ITDG...Oxfam UK&I...Water Aid...ActionAid....CAA Australia, and DfID, UK

WaterAid says "The evaluation selected as an example for 'best practice 1997' isthe participatory evaluation of our Bangladesh programme. Since 1986 WaterAid has been working on integrated water and sanitation projects in rural areas of Bangladesh with the NGO Village Education Resource Centre (VERC). VERC is working in areas of the country that experiences such hydrological problems as saline water intrusion in the south and declining water tables in the north and also with marginalised tribal populations. In 1997 WaterAid set up a country office in Dhaka which caused our relationship with VERC to change from one of donor to partner. The current VERC project is half way through a three year agreement and the mid-term evaluation implemented in February 1998 aimed to:

review WaterAid's rural strategy
to assess the scope of the hygiene education component of VERC's project,
assess the project's use of participatory techniques, especially baseline survey, project planning, monitoring and in the implementation of hygiene education and community development activities, and
comment on the engineering aspects of the project.

It is not yet possible to comment on the final recommendations that have resulted from this work, as the report is still being finalised. However, the evaluation process has been lauded as having been highly effective in raising the profile of participatory practice with the project staff. VERC had no previous experience of participatory evaluations and although they were initially sceptical, were introduced to the new concept by evaluation team member skilled in participatory development. VERC staff were seen to appreciate how this style of evaluation was able to uncover information that would have been missed by a standard non-participatory evaluation. As a result they were won over to the benefits of fully incorporating these techniques in their project work."

More information on this will be able by May 1998 from Jean MacGrory Programme Management and Evaluation Adviser WaterAid - email wateraid@compuserve.com (This item was received 16 April 1998)

Go to responses by: VSO...ITDG...Oxfam UK&I...Water Aid...ActionAid....CAA Australia, and DfID, UK

ActionAid has selected their 1997 Nepal Mid-Term Review as their example of M&E best practice in 1997.

"1.Background ActionAid started to work in Nepal since 1982. It is involved in long term integrated social development programme (micro-projects) and policy research and advocacy work in favour of the poorest and disadvantaged section of societies in different parts of Nepal. Mid-term Review (MTR) of long term programme is a major event in ActionAid Nepal (AAN) which takes place in every five years. In 1997, MTR was carried out in Nawalparasi and Sindhuli districts of Nepal where AAN has been working since 1992. Objective of the review was to see the access of AAN's programme to target groups and changes in communities after the development intervention initiated in 1992. The review also aimed to analyse its sustainability and to workout phase-out strategy for the programme. This review was both process and outcome oriented which took six months to complete. The case cited here is about the MTR in Nawalparasi district in the field of community organisation, education, health, infrastructure development and livelihood and natural resources management.

2. Information about the evaluation

Process The process of MTR took six months to complete (January to July 1997). Unlike other review works where externals are involved, this review sought to engage the targeted people and community in the entire review processes. The MTR process was a composite of several components namely;

Collating information from official documents (use of secondary information)
Utilisation survey
Community review
Thematic study from experts on selected themes
Programme Ranking
Analysis of qualitative change among children, poor, women and Dalit from 1992-97

Highlights of process:. Village profiles (for 12 Village Development Committees) were prepared by AAN staff. In creating a village profile, participatory methods like timeline and social maps were used. Orientation was given to all staff who were to be involved in MTR.

About 200 user groups of the working area including Community Based Organisations (CBOs), Village Development Committees (VDCs), school teachers, children, women, Dalit (untouchables and minority ethnic groups) from poorest households and political representatives reviewed AAN's programmes in absence of AAN staff. Each group developed their own set of indicators and analysed type and extent of intervention, programme achievement and positive and negative consequences it carried in the livelihood of target groups. Not only the programme but also the organisational policy, approach and attitudes and behaviour of development workers and AAN staffs were reviewed by the community (poor, women, children, dalit, CBOs and VDCs). Participatory methods such as social map, mobility map, time line, focus group discussion, case study, ranking, observation and sharing were used throughout the process. Gender discrimination, dalit, people centred development, local institutional development, conflict management, people participation, sustainability, phase out, advocacy and cost analysis were taken as cross-cutting development issues in collection and analysis of information. Reports were also written by respective groups in absence of AAN staff.

After the community review, a utilisation survey was carried out in all VDCs. Case studies were collected from different target groups of women, poor, children and Dalit.

Two thematic studies were carried out during MTR from two external experts. The themes selected for the study were "Poorest Households Focused Programme" and "Community Organisation" and "Community Development Workers" in Nawalparasi and Sindhuli Districts. The study reviewed AAN intervention in these themes. It identified its strengths and weakness and suggested improvement in the future. Similarly, official documents such as Long -Term Perspective (LTP) and programme policies were used as references and financial reports and records were referred for financial review of the organisation.

In addition to programme review, an organisational audit was accomplished by using (7 S) namely system, structure, style, shared value, skill, staff and strategy. Action learning cycle method were used for the organisational audit.

At the end of field work and community review, AAN had around 200 self evaluation reports received from different target groups of the community. Although it was difficult task to reconcile all these reports, AAN accomplished the task. All reports were reconciled and the document presented to community for sharing and comments.

The first document of MTR is published in local language (Nepali) to increase readability and ownership of the document among the target communities. The report is being translated in English for wider circulation in and outside country.

At times the whole MTR process was felt threatening as it is the first of its kind done in Nepal.

Content

MTR report contains information in following areas: .

Changes in target groups: comparison in the condition and position of target groups from 1992 to 1997, data dis-aggregated by poor, women, children, Dalit in the target communities.
Ranking of performance status of VDC level programme for each sector (i.e. health, education, community organisation and institutional development, livelihood and agriculture and infrastructure development).
Programme achievement compared with LTP targets
VDC and sector wise cost analysis (cost allocation) to ensure cost effectiveness in various components such as staff, support and programme.
Community's perception towards AAN programmes.
Utilisation survey report showing the access of target communities in AAN programmes.
Outcome level M&E indicator developed by communities about poverty and effect of AAN programmes.
In-depth study reports on two themes namely "poorest household focused programme (PHFP) and Community organisation (CO).

3. Why is MTR viewed as a good process?

The process used in MTR is viewed as a good practice because of its specific features which ensures peoples participation and ownership to the whole process and output. Such as:

The output contains dis-aggregated information on the basis of social stratification such as poorest households, women, Dalit, children, VDCs.
The MTR did not only look at the programme but also reviewed strength and weakness within the organisation. For this, both programme and internal organisational audit took place.
Community target groups themselves prepared outcome indicator to evaluate the programme. It not only created space for involvement but also increased their ownership to AAN implemented programmes. AAN got a chance to know community perception on policies, approach, staff behaviour, working style etc.)
Qualitative changes between base year and MTR year are clearly observed from community review and utilisation survey even without baseline data.
Progress against target (mentioned in Long-term perspective of the area) are compared and analysed. Presentation of the findings to the people increased their ownership
Report prepared in local language increased readability and widespread circulation within Nepal that can be referred by local NGOs and CBOs.
Cost analysis of programme on the basis of Sectoral programme and VDC has given the cost effectiveness dimension"

For more information contact (yet to be detailed!). This item was received 20 April 1998.

Go to responses by: VSO...ITDG...Oxfam UK&I...Water Aid...ActionAid....CAA Australia, and DfID, UK

CAA have selected "A Review of CAA's Pacific Program" Documentation should be completed and available via this page by May 1998. The contact person is Deborah Elkington

Go to responses by: VSO...ITDG...Oxfam UK&I...Water Aid...ActionAid....CAA Australia, and DfID, UK

A Reply from DfID Evaluation Department:

"MandE NEWS has asked us which of all evaluations you carried out (or funded) in 1997 was the best ?"

"In DFIDs Evaluation Department (EvD), we have not previously made an Evaluation of the Year award and for reasons given below, we feel it would be misguided to do so. However, we are always happy to receive comments and views from readers of DFID-funded evaluations. Nominations for DFID Evaluation of the Year would be received with interest!"

"DFID assists a wide range of development activities. EvD is tasked with evaluating the impact of these activities in a representative way, ensuring that assessment is regularly made in each significant area of work. For this reason, individual evaluation studies fit into a wider, structured evaluation programme spanning several years. Singling out a particular evaluation in a specific year obscures the broader aim of providing a representative view of activities supported by DFID".

"DFIDs approach to evaluation is evolving in response to changes in international development, evaluation practice and readers expectations and interests. While our evaluations were in the past mainly focused on single projects, we now also commission sector synthesis studies drawing on the lessons from relevant summative project evaluations, formative impact assessment work and the evaluation experience of other development agencies. We also commission broader thematic studies; for example, the current work programme includes studies of DFID assistance to such wide fields as environmental protection, poverty reduction, and gender equity. Clearly, these various types of evaluation studies have differing characteristics, purposes and audiences."

"This brief and incomplete sketch of the range of DFIDs evaluation work should provide a clue as to why we are reluctant to settle on any one evaluation as the pick of the crop. These different types of evaluations, with their varied scope and purposes, are not readily comparable. To push the harvesting metaphor a little further, they are apples and pears and consequently defy easy comparison."

"Even within a particular type of evaluation genre, there are serious problems of comparability. Taking the relatively straightforward category of routine project evaluations, one sees immediately that projects lend themselves to evaluation to varying degrees. The varying circumstances of project design and implementation and, in particular, the differing extents to which impacts are traceable make them evaluable to very different degrees. Take the case of a project where a systematic baseline study has been conducted and reported competently, and followed up by further rigorous impact assessment studies during the life of the project. Contrast this case with, say, a cluster of overlapping emergency relief interventions, implemented in a chaotic setting where careful assessment and reporting have not been given priority. Good evaluations could emerge from study of either situation, but their approach would necessarily be very different. One would be hard pressed to justify which was 'best'. "

"Finally, the relevance and usefulness of any particular evaluation report is bound to differ for different readers. In this sense, judgements of quality cannot be universal. What sensible person, after all, would want to nominate Shakespeare's best play? Most DFID evaluations are in the public domain. Rather than imposing our assessment of which are best, we invite readers of our evaluation reports to let us know which they think of the reports we publish."

(by)

Colin Kirk Social Development Adviser, Evaluation Department, DfID (received April 9th, 1998)

Go to responses by: VSO...ITDG...Oxfam UK&I...Water Aid...ActionAid....CAA Australia, and DfID, UK

Since March 26th, 1998 there have been
visits to this page. See MandE Statistics for more details of the usage of MandE NEWS.

Best Evaluation Practice in 1997