Independent Evaluation of Expenditure of DEC Kosovo Appeal Funds

Terms of Reference

Approved - July 22, 1999

H:\ADEC\APPEALS\KOSOVO\Kosovo ToR\Final ToR Kosovo 220799.doc

1 Background

DEC agencies have been present in the Balkans for much of the past decade, including in Kosovo, where assistance has been provided against a background of conflict between Serb and Albanian. The commencement of the NATO bombing of Serbia on March 24^th 1999, following the breakdown of negotiations with President Milosevic, forced many agencies to suspend programmes and withdraw staff from both Serbia and Kosovo.

The bombing of Serbia and the ethnic cleansing of the Kosovar Albanians, resulted in further conflict and displacement in Kosovo. Within just a few weeks, over one million ethnic Albanians were forced to flee their homes, many seeking refuge in neighbouring Albania and Macedonia, as well as in the Yugoslav Republics of Montenegro and Bosnia-Herzegovina. The conflict has also seen many ethnic Serbs displaced from their homes in Kosovo and the rest of FRY.

The international community was unprepared for the scale of the exodus from Kosovo and conditions for the refugees were initially extremely difficult. The possibility that violence would spread to other areas, the involvement of the Western military, and the plight of the refugees, all in a European context, led to massive media interest in the emergency. Against this background, the DEC launched what would become its most successful appeal ever, so far raising over £40 million in pooled funds. A considerable proportion of the initial expenditure of pooled funds, though by no means all, has been channelled to meet the needs of the refugees in Albania and Macedonia. As these refugees have returned to Kosovo, the agencies have re-established programmes in Kosovo to support their reintegration and the rehabilitation of the province.

The refugees have encountered a wide variety of conditions in their countries of asylum: tightly controlled refugee camps; immediate transport to third countries; accommodation with host families; informal camps constructed in civic buildings. Providing assistance to those in need has been complicated by the dispersion of refugees to many sites, the difficulties of obtaining information on those residing with host families, and the weak transport and communication infrastructure in some of the areas hosting refugees. The response has been characterised by the involvement of hundreds of international NGOs, a number of whom had no prior experience in emergency work, nor experience of work in the Balkans. Other important factors in the response have been the presence of NATO troops, acting in a humanitarian role, and the availability of very high levels of funding from bilateral donors.

The Disasters Emergency Committee (DEC) is an umbrella charitable organisation which launches and co-ordinates a National Appeal in the UK in response to a major disaster overseas. It brings together a unique alliance of aid, corporate, public and broadcasting services to rally compassion and ensure that funds raised go to bring effective and timely relief.

The DEC Kosovo Appeal was launched on 6^th April, 1999. To date, the appeal has generated over £40m. pooled funds, which have been disbursed amongst the 12 DEC agencies that participated in the appeal (see Appendix 1 for summary of disbursements). These funds have supported activities in: medicine and health care; water and sanitation; clothing and household items; pre-school activities; education; family tracing, amongst other sectors. (see Appendix 2 for summary of agency activities.)

DEC agencies have been active throughout the region, working in camps, in host communities and with all those affected by the Kosovo crisis.

DEC rules require an independent evaluation of the expenditure of appeal funds to be initiated in the eighth month following an appeal launch. This provides an important mechanism for DEC transparency and accountability to fundraising partners and the British public. Evaluation also enables DEC agencies to extend individual and collective learning on good practice in response to humanitarian emergencies. Once the evaluation process is completed, the final report will be made public.

• Key purpose of the evaluation

The evaluation should provide an independent assessment of the effectiveness and impact of the DEC agencies' response to the Kosovo crisis:

Appropriateness

• Were the actions undertaken appropriate in the context of the needs of the affected population and the context in which the agencies were operating? Was sufficient attention given to the identification of clear objectives and activities that would ensure objectives would be met?

• Was the assistance appropriate in relation to the customs and practices of the affected population?

• To what extent were potential and actual beneficiaries consulted as to their perceived needs and priorities? What was the level of beneficiary involvement in project design, implementation and monitoring? How effective and appropriate were these processes in ensuring relevant and timely project delivery in support of the most needy and vulnerable?

• Was the assistance provided in a timely manner?

Efficiency

• Were resources used efficiently? For instance, were more expensive forms of response (such as air transport) used longer than was necessary? Would greater investment in preparedness measures have resulted in more effective and less costly responses?

Impact

• What direct and indirect evidence is available that the action taken contributed to the reduction of mortality, morbidity and suffering and that the affected population was assisted in maintaining or resuming basic dignity and livelihoods?

• What systems or indicators did the agencies use to evaluate the effectiveness of their work?

Coverage

• Was DEC assistance provided to all major population groups facing life-threatening situations?

• What efforts were made to ensure that particular populations, vulnerable groups and areas were not overlooked?

• Were beneficiaries correctly and fairly identified and targeted?

Connectedness

• Was the assistance provided in a way that took account of the longer-term context?

• Did the assistance seek to strengthen the capacity of local agencies and personnel?

Coherence

• What steps were taken by participating agencies to ensure their responses were co-ordinated with each other and with other agencies?

• Were other actions, such as advocacy work, undertaken by the member agencies to complement their immediate relief actions?

These criteria take into account 'standard' evaluation questions, and also reflect the DEC's Objective, the NGO/Red Cross Code of Conduct and those disaster response objectives of DEC member agencies which are broadly shared. Thus, objectives such as achieving a co-ordinated response, ensuring that relief activities take account of longer-term considerations and that the capacity of local organisations and personnel is strengthened during the response, are explicitly included in the criteria.

Following the field visits the evaluation team should be in a position to comment on the adequacy of management, accounting, monitoring and reporting processes of the DEC agencies and their field level partners. They should also be able to comment on the key constraints that affected the DEC supported programs, and how the different DEC agencies dealt with them.

3 Specific issues for consideration

· What was the added value of DEC appeal funds in the context of the overall humanitarian response? Did DEC funds facilitate a quick response?

· Was there disproportionate attention given to refugees in camps compared to those staying with host families? What more could have been done to support host communities? Since the return of the refugees to Kosovo has a disproportionate level of assistance been focussed on the returning refugees compared to other vulnerable groups both in Kosovo and the other provinces and countries affected?

· Was gender considered in the agencies’ emergency assessments? Did relief provision include special components for women and, if so, were these systematically monitored?

· How did the involvement of NATO troops in supporting humanitarian action affect the co-ordination and delivery of assistance?

· The DEC's mandate is concerned with bringing immediate relief to those in need. However, this appeal, together with funding from bilateral donors, has raised potentially more than can be spent meeting 'immediate' needs. Does this require a re-wording of the DEC mandate? Has the volume of aid funds made available in this response led to a pressure to spend out of proportion with agency capacity or of the needs on the ground? Have standards been too high?

· What was the level of co-operation in the field? Could more have been done to help improve the effectiveness of DEC agencies’ response in terms of joint logistics, communications packages, and inter-agency information flows?

4 Method

Participating DEC agencies will be required to submit the following material (in both hard copy and electronic format) to the Secretariat before the evaluation team commences its work:

· a summary chronology

· key documents on the agency's response to the emergency and their use of DEC funds, including the 48 hour Plan of Action; the 4 week Plan of Action; and the final report of expenditure

· names, contact details and roles during the response of key agency personnel in the head office and in the relevant field offices.

The Secretariat will prepare a package of materials on each participating agency to be given to the evaluation team.

The evaluation team will begin with a review of documentation.

A member of the evaluation team will visit the head office of each agency to undertake preliminary interviews and collect and review supplementary documentation. Evaluators should be allowed full access to relevant files. The schedule of the fieldwork will be discussed during these visits. This is likely to be a difficult issue give that many operations will already have closed down by the time the evaluation team is ready to begin fieldwork. The consultants schedule, accommodation and transport arrangements will be finalised and communicated to all agencies at least one week prior to the team's visit.

In the field the evaluation team will seek to spend a period with each agency which is roughly proportional to the share of DEC pooled funds received by each agency. During their work the evaluators will fill out the chronology of decisions and actions so as to understand the context and the level of information that was available to the agency in deciding on a particular action. During their time with each agency the team will interview key personnel remaining in-country (contacting others prior to the field visits or on their return) and undertake visits to selected project sites/areas. It is recognised that many of the sites will have closed down by the time the evaluators reach the field. The evaluators will have to make extensive use of agency reports where site visits would prove pointless. It should be noted that in the case of agencies that are part of larger organisations UK assistance might not be distinguishable from that of global counterparts.

As well as interviewing the agency's project officers and key officials in co-ordinating agencies (eg. OCHA, UNHCR and the host government) and partner agencies, a sample of beneficiaries will be selected and interviewed by the evaluators. These interviews will be conducted without agency personnel being present using interpreters hired directly by the evaluation team. The beneficiaries will be questioned on their views of the assistance provided, the way they were selected and the assistance provided and their overall views of the agency. Interviews with individuals may be complemented by discussions with groups of beneficiaries. So as to assess the agency's targeting and beneficiary selection methods the evaluation team will also interview a selection of potential beneficiaries who did not receive assistance. Given the fact that many of the camps will be closed by the time the evaluators visit, related beneficiaries will be interviewed in Kosovo.

It is expected that the evaluation team will use gender-aware and participatory approaches to seek the views of beneficiaries and, where appropriate, non-beneficiaries. Inclusive techniques will be expected of the evaluators, to seek active participation in the evaluation by members of local emergency committees, staff of implementing partner agencies and member agencies, representatives of local and central governments.

Agencies ‘Final Expenditure Reports’ will be examined to assess direct and indirect project costs, and, in conjunction with beneficiary/team assessment of direct and indirect benefits, and to compare the cost-effectiveness of different approaches.

The evaluation will be undertaken with due reference to the Red Cross/Red Crescent NGO Code of Conduct, which all agencies have signed. Reference should also be made to the Sphere Standards.

Before leaving the country members of the team will indicate their broad findings to the Country Representative and senior staff of each agency and note their comments.

There is currently an on-going DEC lesson-learning study taking place in respect of the Balkans emergency. Two data collectors have spent time in the region conducting interviews and collecting key documents. The principal focus of the lesson-learning study has been the issue of co-ordination. The evaluation team will be able to draw upon the findings and documentation collected by the lesson-learning team.

A workshop should be held in London to disseminate a draft report of the evaluation, the report should be circulated one week prior to the workshop to allow for preliminary comment.

5 The Report

The evaluation report should consist of:

• executive summary and recommendations (not more than six pages)
• main text, to include emergency context, evaluation methodology, appeal management, commentary and analysis addressing evaluation purpose and outputs, conclusions (not more than thirty pages)
• appendices, to include evaluation terms of reference, maps, sample framework, summary of agency activities, sub-team report(s), end notes (where appropriate) and bibliography.

6 Evaluation team and timeframe

Given the speed with which many of the DEC-funded programmes in the region have closed, as refugees return to Kosovo, it is anticipated that there will be a core team of three people, with others drawn in as necessary. The Team Leader should be a generalist with a proven background in emergency evaluations. The appropriate balance of professional and analytical skills amongst the remaining team members should be determined following a preliminary examination of agency activities. It is likely, however, that expertise in water and sanitation and public health will be required. At least one person from the region should be included in the team that makes the field visits. All team members should be gender aware, a gender balance within field teams being desirable.

Consultants or independent evaluation teams short-listed in the tendering process should seek DEC approval for any proposed changes to the composition of the team originally submitted.

The evaluation timeframe should allow for circulation of a first draft, for comment by DEC agencies, in early May 2000 followed by presentation of the draft report to member agencies by end of May for completion by 10^th June 2000.

7 Tenders and Evaluation Management

Tenders should be submitted to the DEC Secretariat by the closing date of 23^rd August 1999. A maximum 5 page summary should be submitted with appendices of team member CVs and an indication of availability. The DEC may wish to see substantive pieces work or to take up references of short-listed consultants.

The final decision on tenders will be taken by the DEC Executive Committee, following short-listing and interviews. Key factors will include:

Provisional framework, methodology, team balance, local experiences, distinctive competencies, timeframe and budget, an appreciation of key constraints and comments on the above terms of reference.

Professionalism of the bid, team experience (professional and analytical), degree of parity with the terms of reference, likelihood of achieving the DEC timetable, realism, not just competitiveness, in the cost submission.

Tenders will be accepted from "freelance" as well as from company, PVO or academic teams.

Administration and overall co-ordination, including monitoring progress, lies with the DEC Secretariat. The evaluation Team Leader must, from the commencement of the evaluation, submit a weekly report on actual against anticipated progress. The Steering Committee will via the Secretariat undertake to respond to weekly submissions as necessary. In addition, the Team Leader should alert the Secretariat immediately if serious problems or delays are encountered. Approval for any significant changes to the evaluation timetable will be referred to the DEC Operations Sub-Committee.

It is anticipated that the selection process will be completed by mid June.

9 Further information

For further information please contact:

Jamie McCaul, Executive Secretary

DEC

52, Great Portland Street

London, W1N 5AH

Tel. 44 + (0)171 580 6550

Fax. 44 +(0)171 580 2854

email decuk@ compuserve.com

Or visit the DEC’s website: www.dec.org.uk

DEC ‘Central America Hurricane Appeal’

Funds Disbursed to Member Agencies

* Funds are returned for reallocation where agencies are unable to spend their due share or where appeal moneys remain unspent after 6 months. This ensures that remaining sums are reallocated to projects that most closely meet the DEC’s relief mandate. This does not prevent funds being reallocated to agencies that have returned funds.

Return to Mande NEWS