Comments by Rick Davies on:

Rights, Culture And Contested Modernities.

A paper presented by David Marsden, Tuesday 1^st April 2003 at INTRAC's 5th International Evaluation Conference: Measurement, Management, and Accountability?, KFK Conference Centre, The Netherlands. Details available at http://www.intrac.org/Intrac/5thEvaluationConference_en.html

These are based on notes taken during the presentation, which I then responded to. Given the breadth of David's paper I have not tried to respond to every issue, just those that concerned me the most, at the time

• The post-modern perspective. This is where David comes from
  • How can it help, is the question most people will be asking. The relativism resonates but also worries some people
  • It is possible to develop some form of "practical post-modernism". The "Most Significant Changes" approach (www.mande.co.uk/docs/ccdb.htm) has no single predefined definition of what is desirable, and the process of selecting and defining what futures are most desirable is situated within an explicit power context, but one which nevertheless gives the less powerful significant capacity to define the range of choices

• Culture: everyone recognises its importance as emphasised by David, I think. The question again is how do we respond to culture when designing and managing M&E systems.
  • My suggestions relate to two possible responses
    • Seeing all development activities in terms of networks of actors, not just linear sequences connecting our actions to the lives of poor people, along a chain of intermediaries.
      • This is one way of making the context of our activities more visible, by locating ourselves in relationships to other actors and their relationships
      • See my Ad Lib presentation on Wednesday where this idea is expanded (www.mande.co.uk/docs/adlib.htm) .
      • See also my paper on representing theories of change at www.mande.co.uk/docs/RDseville2.doc which includes a discussion of networks.
    • Making much more use of ranking forms of measurement. This is the simplest form of measurement available. Yet it is under utilised.
      • Anyone can make ranking judgements. And people do make ranking judgements every day, when they decide to do x rather than y
      • Pair comparisons of ranked entities can pull out criteria of concern on an inductive basis, grounded in people's experience to date.
      • In doing so what we are doing here is making existing informal and tacit knowledge more formal (documented) and explicit. A lot of cultural knowledge exists in informal and tacit knowledge. Making it explicit and formal enables more conscious and public choices about what aspects of that cultural knowledge are of most value in the circumstances, and in the future under consideration.
      • Ranking methods acknowledge the complex nature of human judgements. That is that judgements can be made using a number of criteria of concern at the same time. Not just one single measure.
      • Combinations of ranking judgements (e.g. ranked priority of objectives versus ranked achievement of objectives) can produce informative representations of both trends, and outliers from trends, which are useful for both accountability and learning needs respectively.

• David's paper mentions a number of dichotomies, such as mechanistic versus organic. These are undoubtedly useful for rhetorical purposes, helping to empathise points of difference. Robert Chambers has made a lot of use of opposites as a way of illustrating what he thinks is the way we should go.
  • The downside is that these dichotomies can end up limiting our choices about how we are do things. Linear, rational, mechanistic methods become a bad thing, and we now have to do (presumably) the opposite. We just swap sides in a debate but end up having no more net room to move than before.
  • The alternative is for us to try to think about both ends of the spectrum and ask where and when we should use which approach. For example, where and when top-down versus bottom-up systems are most useful. [But we did not manage to get that far in the conference. Instead we seemed to have got stuck on top-down is bad, bottom-up is good]
  • Despite having developed a very inductive and participative monitoring system (Most Significant Changes) I also think there is a big place for deductive and rationally developed "Theory of Change" approaches to M&E. [These were conspicuously absent from discussions throughout the workshop, but they can be very useful]. I like to use both inductive and deductive approaches
  • Another example of the bad effects dichotomous thinking of is the too casual dismissal of before/after and with/without comparisons in the evaluation of social development programmes because they are seen to be part of a rationalist scientific approach. And then no thought to what else can be done in place. There is an alternative, especially in large programmes. That is to make comparisons of internal differences in responses to programme interventions by different groups assisted by development programmes. SCF (US)'s work on positive deviance within child nutrition programmes in Vietnam is probably the most well publicised example of this approach.

• The dismissal of the importance of measurement. This came across in David's paper [and subsequently in much of the discussion during the conference]
  • We need to remind ourselves that people do make judgements, all the time, every day. The question is whether we want to make those judgements visible, and open to questioning. If so, then ranking measures to start with, then other more demanding forms of measurement, can be very appropriate. But the reverse, of lots of measurement activities, without explicit and accountable interpretation of their meaning can be equally problematic.
  • Reference was made to the idea of "communities of practice", an idea that links back to the idea of a networked perspective on development activities. One which resonates with my own views. The downside is that we may simply be re-labelling our enduring myths and hopes about the world. That there is a cohesive and mutually supportive community somewhere out. If not in third world villages, then maybe amongst us as development workers or scholars. This is where the usefulness of measurement and empirical research comes into play. So far the research on academic networks shows that within "communities of practice" there are highly unequal patterns of interaction between participants, with some actors having large numbers of links and many others having only a few. The same structure is found on a more macro level, within the structure of the World Wide Web (Barabasi, 2002). This knowledge then presents us with a choice as to how we should respond

• Scaling up and replication were also given a bad press. Not seen as desirable things or certainly as expectations to be sceptical about.
  • This should not be surprising to hear from an anthropologist, whose professional remit is in effect to celebrate the local and unique, and diversity in general. And there is a lot to be said for that perspective in a world that is seeing the rapid extinction of species and cultures on a scale never seen before (at least since the Jurassic period).
  • But it does provoke some big questions about what aided development is all about.
    • Is it an art, where every achievement is (ideally) uniquely valuable and almost by definition non-replicable.
    • Is it a craft, where all we can expect at best is that what apprentices will learn are various rules of thumb and homilies that will give general guidance The rest must be learned informally and tacitly by prolonged exposure to the work of others who have had more experience?
    • Is it a science, where valuable achievements are precisely those that are replicable, but which also add something new to the sum total of human knowledge. Here explicitly documenting how things are done (and not just their outcomes) is a core part of the process. Here there is a premium on disembodied knowledge, because it must be abstract enough to be able to be transmitted and used in different contexts.
  • Trust was mentioned by David (and others during the conference) as the alternative to demanding M&E systems. O'Neils 2002 lecture on this topic was mentioned. This makes sense if the focus of M&E systems is on accountability. But it makes no sense if we expect development projects to generate some generalisable knowledge that can be re-applied elsewhere. What we need to do instead is develop M&E systems that produce usable knowledge, that is picked up and re-applied by others elsewhere. In doing so they will help make the original investment in the first project much more cost effective, and give us a more solid basis for advocating increased investment in aided development. It is not evidence of impact we need, but replicable knowledge of how to re-create such impacts.

(These notes were written up on April 11th, 2003)

For more information on Rick Davies perspective go to the Editorial section of Monitoring and Evaluation NEWS and Rick Davies on the Internet