Return to Monitoring and Evaluation NEWS

Independent Evaluation of
Expenditure of DEC Mozambique Floods Appeal Funds

Terms of Reference

(18/04/2000 – Approved)

1 Background

Around 8^th February 2000, Mozambique, one of the world’s poorest countries, suffered its worst floods for half a century. Following the unusually heavy rains over the region and the tropical storms that accompanied cyclones Connie and Eline, hundreds of square miles around the Limpopo and Save river basins were left under water, with hundreds of thousands of people homeless and at risk.

The Mozambique Government estimated that up to 1 million people had been affected with hundreds feared dead. Overwhelmed by the devastation it appealed to the international community for assistance.

The Disasters Emergency Committee (DEC) is a UK charity which launches and co-ordinates a National Appeal in response to a major disaster overseas. It brings together a unique alliance of aid, corporate, public and broadcasting sector services to rally compassion and ensure that funds raised go to those amongst its fourteen UK member aid agencies best placed to bring effective and timely relief. DEC agencies have been present in Mozambique for many years working directly or supporting local partners.

The DEC ‘Mozambique Floods Appeal’ was launched on 2^nd March 2000. To date, the appeal has generated some £20m pooled funds to be disbursed to the 11 DEC agencies participating in the appeal. These funds are supporting activities in: search and rescue; water and sanitation; food aid; medicine and health care; clothing and household items; shelter and housing; seeds and tools etc (see Appendix 2 for summary of agencies’ activities and locations). There will be an initial nine-month expenditure period following the launch of this appeal, during which participating agencies will have access to a pre-determined share of pooled appeal funds (see Appendix 1 for summary of disbursement shares). Participating agencies will submit a ‘Final Declaration of Expenditure’ reports in the tenth month following the launch.

DEC rules require an independent evaluation of the expenditure of appeal funds. This provides an important mechanism for DEC transparency and accountability to fundraising partners and the British public. Evaluation also enables DEC agencies to extend individual and collective learning on good-practice in response to humanitarian emergencies. The final report will be made public on completion of the evaluation.

2. Main purpose of the evaluation

To provide independent assessment of the effectiveness and impact of the DEC-funded responses following the Mozambique floods.

Appropriateness

• Were assessments undertaken appropriate to identification of need? Were the actions undertaken appropriate in the context of the needs of the affected population and the context in which the agencies were operating? Was sufficient attention given to the identification of clear objectives and activities that would ensure objectives were met?
• Was the assistance appropriate in relation to the customs and practices of the affected population?
• To what extent were potential and actual beneficiaries consulted as to their perceived needs and priorities? What was the level of beneficiary involvement in project design, implementation and monitoring? How effective and appropriate were these processes in ensuring relevant and timely project delivery in support of the most needy and vulnerable?
• Was the assistance provided in a timely manner?

Efficiency

• Were resources used efficiently? For instance, were more expensive forms of response (such as air transport) used longer than was necessary? Would greater investment in preparedness measures have resulted in more effective and less costly responses?

Impact

• What direct and indirect evidence is available that the action taken contributed to the reduction of mortality, morbidity and suffering and that the affected population was assisted in maintaining or resuming basic dignity and livelihoods? In the absence of much baseline data, it is suggested this might best be measured against the level of satisfaction of beneficiaries and their perception of appropriateness and effectiveness of the response.
• What systems or indicators did agencies use to evaluate the effectiveness of their work?

Coverage

• Was DEC assistance provided to all major population groups facing life-threatening situations?
• What efforts were made to ensure that particular populations, vulnerable groups and areas were not overlooked?
• Were beneficiaries correctly and fairly identified and targeted?

Connectedness

• Was the assistance provided in a way that took account of the longer-term context?
• Did the assistance seek to strengthen the capacity of local agencies and personnel?

Coherence

• What steps were taken by participating agencies to ensure their responses were co-ordinated with each other and with other humanitarian agencies?
• Were other actions, such as advocacy work, undertaken by the member agencies to complement their immediate relief actions?

These criteria take into account 'standard' evaluation questions, and also reflect the DEC's Objective, the NGO/Red Cross Code of Conduct and those disaster response objectives of DEC member agencies that are broadly shared. Thus, objectives such as achieving a co-ordinated response, ensuring that relief activities take account of longer-term considerations and that the capacity of local organisations and personnel is strengthened during the response, are explicitly included in the criteria.

Following the field visits the evaluation team should be in a position to comment on the adequacy of management, accounting, monitoring and reporting processes of the DEC agencies and their field-level partners. They should also be able to comment on the key constraints that affected the DEC supported programs, and how the different DEC agencies dealt with them.

3 Specific Issues for Consideration

· What was the added value of DEC appeal funds in the context of the overall humanitarian response? Did DEC funds facilitate a quick response?

· Was gender considered in the agencies’ emergency assessments? Did relief provision include special components for women and, if so, were these systematically monitored?

· Were the special needs of acute vulnerable groups (e.g. children/elderly/disabled etc) considered in the agencies’ emergency assessments and were they consulted in the same way as other groups? Did relief provision include special components for them and if so were these appropriate and systematically monitored?

· Did the response of the DEC agencies strengthen and complement the response of local organisations and coping mechanisms, or hinder them?

· What was the level of co-operation in the field? Could more have been done to help improve the effectiveness of DEC agencies’ responses in terms of co-ordination, joint-logistics, communications packages, and information flows between the key relief players?

• Was there appropriate geographical coverage within the affected region?
• To what extent did responses reflect lessons-learned from previous flood disasters?
• To what extent did the DEC agencies’ limited involvement in ‘search & rescue’ and the difficulties around transport affect the impact of the DEC funded response? The evaluation team is requested to dedicate one section of the final evaluation report to this issue.

4 Method

Participating DEC agencies are required to submit the following material (in both hard copy and electronic format) to the Secretariat to assist the evaluation team’s work:

· a summary chronology

· key documents on the agency's response to the emergency and their use of DEC funds - e.g. ‘48 Hour Plan of Action’; ‘4 Week Plan of Action’.

· names, contact details and roles during the response of key agency and partner personnel in the head office and in the relevant field offices.

• List of indicators used by the agencies to monitor and evaluate their DEC funded activities.

The Secretariat will prepare a package of materials on each participating agency to be given to the evaluation team, as well as minutes of appeal related decision-making meetings – e.g. decision to appeal; decision to extend/reduce the period of joint action; decisions affecting DEC rules governing appeals.

The evaluation team will begin with a review of available documentation.

The evaluation team will be responsible for ensuring appropriate data-collection is undertaken in the field following their appointment, so that key information, that may no longer be available in the later stage of the DEC funded response, is not lost to the evaluation process.

Following submission of DEC agencies’ ‘10th Month Declaration of Expenditure’ reports, member(s) of the evaluation team will visit the head office of each participating agency to undertake interviews and collect and review supplementary documentation. Evaluators should be allowed full access to relevant files. The schedule of the subsequent fieldwork will be discussed during these visits. Since certain operations will already have closed down by the time the evaluation proper is underway, it will be appropriate to undertake preliminary fieldwork during the expenditure period. The evaluation team’s schedule, accommodation and transport arrangements will be finalised and communicated to the Secretariat and all agencies at least one week prior to any visit.

In the field the evaluation team will seek to spend a period with each agency that is roughly proportional to the share of DEC pooled funds received by each agency. During their work the evaluators will fill out the chronology of decisions and actions so as to understand the context and the level of information that was available to the agency in deciding on a particular action. During their time with each agency the team will interview key personnel remaining in-country (contacting others prior to the field visits or on their return) and undertake visits to selected project sites/areas. The field visit must include at least one DEC funded project for each participating agency. The evaluators will have to make extensive use of agency reports and their own preliminary data collection, where later site visits would prove pointless. It should be noted that in the case of agencies that are part of larger organisations UK assistance might not be distinguishable from that of global counterparts, nevertheless, every effort should be made to distinguish DEC funding.

As well as interviewing the agencies' project officers, key officials in co-ordinating agencies (e.g. UNICEF, OCHA, central and state governments), and partner agencies, a sample of beneficiaries will be selected and interviewed by the evaluators. These interviews will be conducted without agency personnel being present, using interpreters (where necessary) hired directly by the evaluation team. The beneficiaries will be questioned on their views of the assistance provided, the way they were selected and their overall views of the agency. Interviews with individuals may be complemented by discussions with groups of beneficiaries. So as to assess the agency's targeting and beneficiary selection methods the evaluation team will also interview a selection of potential beneficiaries who did not receive assistance.

It is expected that the evaluation team will use gender-aware and participatory approaches to seek the views of beneficiaries and, where appropriate, non-beneficiaries. Inclusive techniques will be expected of the evaluators, to seek active participation in the evaluation by members of local emergency committees, staff of implementing partner agencies and member agencies, and representatives of local and central governments.

Agencies’ ‘10^th Month Declaration of Expenditure’ reports will be examined to assess direct and indirect project costs, and, in conjunction with beneficiary/team assessment of direct and indirect benefits, and to compare the cost-effectiveness of different approaches.

The evaluation will be undertaken with due reference to the Red Cross/Red Crescent NGO Code of Conduct, which all agencies have signed. Reference should also be made to the Sphere Standards.

Before leaving the country, members of the team will indicate their broad findings to Country Representative and senior staff of each agency and note their comments.

A meeting should then be held in London to disseminate a draft report of the evaluation. The report should be circulated one week prior to the workshop to allow for preliminary review by agencies and their partners, and followed by a two-week formal agency comment period.

The evaluation team should allow for a follow-up workshop in-country within a month of the release of the final evaluation report. The aim of this workshop will be to discuss the evaluation recommendations and major lessons of the Mozambique floods, and how agencies might seek to implement. It is suggested that participants include UK, regional, and in-country representatives from the agencies and their implementing partners, and other key stakeholders as appropriate.

5 The Report

The evaluation report should consist of:

• executive summary and recommendations (not more than six pages)

• main text, to include index, emergency context, evaluation methodology, appeal management, commentary and analysis addressing evaluation purpose and outputs to include a section dedicated to the issue of particular lesson-learning focus, conclusions (not more than thirty pages)

• appendices, to include evaluation terms of reference, maps, sample framework, summary of agency activities, sub-team report(s), end notes (where appropriate) and bibliography. (All material collected in the undertaking of the evaluation process should be lodged with the Secretariat prior to termination of the contract)

6 Evaluation team and timeframe

It is anticipated there will be a core team of at least three people, with others drawn in as necessary. The Team Leader should have a relevant skill and a proven background in emergency evaluations. The appropriate balance of professional and analytical skills amongst the remaining team members should be determined following a preliminary examination of agency activities. It is likely, however, that sector expertise in areas such as water and sanitation, public health and shelter will be required. At least one person from the region should be included in the team that makes the field visits.

All team members should be gender aware, and a reasonable gender balance within field teams is desirable.

Consultants or independent evaluation teams short-listed in the tendering process should seek DEC approval for any proposed changes to the composition of the team originally submitted.

The evaluation timeframe should allow for the circulation of a first draft by early March 2001, followed by presentation of the draft by the evaluation consultant(s) to member agencies a week later. A formal comment period, of at least two weeks, for participating agencies and their partners will then follow. The completion date for the Final Evaluation Report will be 15^th April 2001, the consultants having addressed agencies’ comments as appropriate.

7 Tenders and Evaluation Management

Tenders should be submitted to the DEC Secretariat by the closing date of 15^th May 2000. A maximum 5 page summary should be submitted with appendices of team member CVs (each CV a maximum of 3 pages) and an indication of availability. The DEC may wish to see substantive pieces work or to take up references of short-listed consultants.

The final decision on tenders will be taken by the DEC Executive Committee, following short-listing and interviews. Key factors will include:

Provisional framework, methodology, team balance, local experiences, distinctive competencies, timeframe and budget, an appreciation of key constraints and comments on the above terms of reference.

Professionalism of the bid, team experience (professional and analytical), degree of parity with the terms of reference, likelihood of achieving the DEC timetable, and realism not just competitiveness in the cost submission.

Tenders will be accepted from "freelance" teams as well as from company, PVO or academic teams. Tenders are particularly welcome from regional teams.

Administration and overall co-ordination, including monitoring progress, lies with the DEC Secretariat. The evaluation Team Leader must, from the commencement of the evaluation, submit a weekly report on actual against anticipated progress. The Steering Committee (DEC Operations Sub-Committee) will via the Secretariat undertake to respond to weekly submissions as necessary. In addition, the Team Leader should alert the Secretariat immediately if serious problems or delays are encountered. Approval for any significant changes to the evaluation timetable will be referred to the Steering Committee.

It is anticipated the selection process will be complete by beginning June 2000.

6 Further information

For further information please contact:

Jamie McCaul, Executive Secretary

Disasters Emergency Committee (DEC)

52, Great Portland Street

London, W1N 5AH

Tel. 44 + (0)171 580 6550

Fax. 44 +(0)171 580 2854

email decuk@compuserve.com

Or visit the DEC’s website: www.dec.org.uk

DEC ‘Mozambique Floods Appeal’

* This is the predetermined percentage share of pooled appeal funds to which each DEC participating agency has a right during the initial 9-month expenditure period following the launch of the Mozambique appeal. The evaluation should, however, use sums declared as spent in agencies’ ‘10^th Month Declaration of Expenditure Reports’ for the purposes of any financial analysis.

DEC ‘Mozambique Floods Appeal’ - Overview of Participating Agencies’ Activities (16/04/2000)

NB The above summary is drawn from agencies’ ‘4 Week Plans’ and has not been verified by the agencies. Not all activities outlined will necessarily be DEC-funded.

Return to Monitoring and Evaluation NEWS