Frequently Asked Questions (FAQs) re

Monitoring and Evaluating Success in Katine

This page is edited by Rick Davies, and was last updated on: 11th January 2008

• Documents provided to the external evaluator by AMREF & Barclays (updated 11/01/08)
• Reports produced by the external evaluator (updated 21/12/07)
• Recommendations made by the external evaluator (updated 19/12/07)

Evaluation criteria

1. What criteria will be used to assess the performance of the Katine project?
• It is not practical  to assess a project on innumberable criteria, not is it fair to assess a project on criteria that are not agreed upon in advance. This FAQ lists some criteria that will be subject of discussion, and possible agreement, with AMREF by December 2007. The final agreed list of assessment criteria will be informed by at least three sources of ideas
1. The OECD Development Assistance Committee (DAC) evaluation criteria, which are widely used in the evaluation of development aid programs. The five DAC criteria are: Relevance, Effectiveness, Efficiency, Impact, and Sustainability. These are all discussed below
2. Comments and questions posed on the the Katine Chronicles blog. You can see some of these below, indicated by their Comment number and link.
3. Responses made to an online evaluation stakeholders survey that will be undertaken in November 2007. The results of this survey will be available via a link to be posted here in late November. In the first instance, this survey will focus on the organsiations who are contractually involved within the UK, and within Uganda. (Community level consultations will be the responsibility of AMREF).

Relevance

1. Have you asked the people of Katine what they want? And I do not mean consultations between an ngo and some people called to a meeting. Believe it or not market research is available in Africa. Comment No. 748913
1. In my January 2008 visit one of my tasks will be to examine the ways in which AMREF staff have worked with people in Katine to identify their needs and priorities. My expectation is that they will use a variety of means, possibly including one to one surveys.

2. Why is the "independent auditor" not African? Is it because his responsibility is to the Guardian and its readers, and not to the people of Katine? Comment No. 748913
1. I will not be the only person who will be monitoring and evaluating the performance of AMREF's work. In the first instance AMREF will have its own dedicated staff and procedures for monitoring and evaluating the implementation and outcomes of its work. There may also be community level committee of local stakeholders, with similar responsibilities. My role will be more "meta-monitoring and evaluation". One of my tasks will be to check on the adequacy of AMREF's systems and procedures for monitoring and evaluating its work. And to provide to them, where wanted, with help to build their M&E capacity. My work will involve talking to AMREF staff, reading their documents, and talking to people in the community. In the longer term, my role should become less and less important, as the Guardian and its readers feel increasingly confident in the abilities of AMREF.

3. What do you mean by relevance?
• The DAC website view is "The extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor. In evaluating the relevance of a programme or a project, it is useful to consider the following questions:
  
  • To what extent are the objectives of the programme still valid?
  • Are the activities and outputs of the programme consistent with the overall goal and the attainment of its objectives?
  • Are the activities and outputs of the programme consistent with the intended impacts and effects?"

Efficiency

1. What do you mean by efficiency?
• The DAC website view is "Efficiency measures the outputs -- qualitative and quantitative -- in relation to the inputs. It is an economic term which signifies that the aid uses the least costly resources possible in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted. When evaluating the efficiency of a programme or a project, it is useful to consider the following questions:
  
  • Were activities cost-efficient?
  • Were objectives achieved on time?
  • Was the programme or project implemented in the most efficient way compared to alternatives?"
• Okay, but what is an output?
• Products or services that result from the project managers (e.g. AMREF ) activities, and which other people (e.g people in Katine) can use. So, re-organising the office files is an activity, but not an output, because the people of Katine would not be able to make any direct use of the re-organised files. But running a workshop on government policy on edcuatio costs is an activity that AMREF could organise, and which the people of katine could particpate in. So, it could be seen as an output
  

Effectiveness

1. Why are the KPI's of the project so vague? Why not use simple KPI's like average income, child survival rates, deaths per thousand. Is it because nice and vague KPI's allow a white wash of the project. Comment No. 748913
• The objectives need to be agreed on first, through community consultations. And then in the process, verifiable indicators of their achievement need to be identified. Again, this is best done through consultations with the groups / communities involved

2. What do you mean by effectiveness?
• The DAC website view is  "A measure of the extent to which an aid activity attains its objectives. In evaluating the effectiveness of a programme or a project, it is useful to consider the following questions:
  
• To what extent were the objectives achieved / are likely to be achieved?
• What were the major factors influencing the achievement or non-achievement of the objectives?"

Impact

1. What do you mean by impact?

• The DAC website view is "The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. The examination should be concerned with both intended and unintended results and must also include the positive and negative impact of external factors, such as changes in terms of trade and financial conditions. When evaluating the impact of a programme or a project, it is useful to consider the following questions:
  

  • What has happened as a result of the programme or project?
  • What real difference has the activity made to the beneficiaries?
  • How many people have been affected?"

Sustainability

1. But this brings up the question in what will happen to Katine in 4 years, 10 years, 20 years after the Guardian campaign has ended. Africa is littered with 'Aid Experiments' that have failed. And to successfully develop somewhere takes decades not 3 years. Comment No. 748913
• I will suggest that when the current "project" comes to an end in 2010 that AMREF tries to make a number of verifiable predictions about what will happen to the various achievements to date, over the next three years (i.e. by 2013). This does not mean we expect AMREF staff to be  clairvoyant. Just that they try to make a reasonable best guess assessment of what is likely to happen
• I have asked the Guardian to consider committing myself (or another independent evaluator) to a follow-up visit and review of Katine three years after the end of the current project (i.e. in 2013). As well as looking for unexpected changes, that visit should also try to assess to what extent AMREF's predictions have been correct or not.

2. What do you mean by sustainability?
• The DAC website view is " Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable. When evaluating the sustainability of a programme or a project, it is useful to consider the following questions:
  
• To what extent did the benefits of a programme or project continue after donor funding ceased?
• What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project?"

Equity

1. Having read AMREF's structured programme to tackle the issues of health, education, sanitation and safe water, income generation and good governance I am concerned to see there appears to be no specific acknowledgment of the centrality of women to the success or otherwise of the outcomes in each area. Comment No. 748693
• One of the criteria that I will be using to assess the work of AMREF will be how they address issues of equity. And one important dimension of equity is gender equity. I will examine not only how women benefit, versus men, but also how they are involved in planning, implementation and review processes

2. Also I urge Mr Davis when he is making his assessments on the project to ensure he is able to speak to women and girls in the right environment - they should have an opportunity to speak in the absence of their men folk whose presence may inhibit them discussing real issues. And when he feeds back he should be careful to ensure individual women are not identified as raising particular issues for fear of retribution - domestic violence is an issue there as it is anywhere else in the world. Comment No. 748606
• In the first instance I will try to identify to what extent AMREF staff have been able to do this. Or if not, why not. Secondly, I will seek to follow this advice myself, when in Katine.

3. What do you mean by equity?
• Equity is the concept of idea of fairness or justice . In a development project there can be equity (or inequity) during the project  implementation process, and later on in the effects that the project has on people's lives. When evaluating the equity dimension of a project it will be useful to consider
• Who has, and has not participated in planning and consultation processes (at various stages of the project's life), about the various developments that have been supported?
• Who has paid, and not paid, what costs (in time, money or kind) towards the implementation of the project?
• Who has benefited from  the project , in both the short and long ter
• And "who" needs to be thought about in terms of multiple kinds of differences that may be important: gender, age, wealth, class, caste, ethnicity, nationality, etc.  

Transparency

1. I would also like this project to be very clear and transparent on how much of the money (cash) raised is actually spent on the people in Katine i.e. how much pence of each pound raised. For the last 25/30 years Uganda's been awash with Government and non-governmental NGOs who receive millions of pounds and yet we see precious little reaching the people in whose name it is all being raised for. Sad to say but charity is a very big lucrative business.Comment No. 748529
1. I have suggested to AMREF that the prepare a Disclosure Policy, which would state what types of information will, by default be publicly available, and what will not, but which may be available on request. Disclosure Policies are used by major organisations such as the World Bank, and the IMF, but also by progressive NGOs such as ActionAid
2. Costs can be cut horizontally (e.g. the amounts spent in UK, spent in Kampala, and spent in Katine) or vertically (the total costs incurred at all levels for delivering a specific outcome on the ground e.g. building a new school or assisting a specific community). The latter is more useful, if you want to compare costs against benefits. Analysing costs horizontally can involve mistaken assumptions:  that all assistance will be in the form of things or money given directly to people in Katine, and that purchases made in Kampala or London will be of no benefit to them.  

2. Avoiding the corruption trap. How can those of us who have chosen to support this project be assured that our contributions will not fund corrupt practices and bribes as we saw on Channel 4's Dispatches last night? Comment No. 762592 October 30 17:32
•  AMREF already have procedures for internal and external auditing of their finances (both management and project related). I will not duplicate this work. I will however encourage AMREF to include reference to the public availability of these reports, in the Disclosure Policy that I have recommended they develop. I will also pay attention to any recommendations made by the Auditor , to see if they have been implemented or not (31/10/07)

3. What do you mean by  transparency?
• "The disclosure of information such as budgets, reports, and financial statements, in order to promote accountability and openness within corporations, governments, and organizations."
• Transparency is "a requirement that sits alongside ‘Accountability’ as a growing expectation on organisations by society. It implies an openness and willingness to accept public scrutiny that diminishes the capacity for an organisation to practice or harbor deception or deceit"

Glossary