The Management of Diversity in NGO Development Programmes

THE MANAGEMENT OF DIVERSITY IN NGO DEVELOPMENT PROGRAMMES

A paper for the Development Studies Association Conference, Dublin, September 1995.

Rick Davies, Centre for Development Studies, Swansea, E-mail: rick@shimbir.demon.co.uk, Phone/fax: 44 (0)1792 203815. URL: http://www.swan.ac.uk/cds/rd1.htm

INTRODUCTION

The word "manage" can mean to cope with, to control or to make masterful use of. All these interpretations can be applied to how NGOs manage the diversity inherent in their development activities. The focus of this paper is on how diversity is managed within the context of monitoring and evaluation systems. The paper looks briefly at some of the forms of diversity that can be generated and the forces generating diversity, and then in more detail at a range of responses to that diversity, ranging from the traditional, to reformist, to the more innovative. The possibilities moving from coping and control oriented responses to more positive responses that make full use of diversity are explored.

The empirical basis of this paper is the experience of working with two Bangladeshi NGOs during the last two years, Proshika and the Christian Commission for Development in Bangladesh (CCDB).

FORMS OF DIVERSITY IN NGO PROGRAMMES

At least five types of diversity can be identified:

The people involved in development programmes (differentiated by age, gender, class, ethnicity, location, preferences etc.)
The different activities that individuals undertake with the use of specific forms of NGO assistance (e.g the whole range of business ventures aided with credit, the uses of grants, and training).
The outcomes that can result from those activities.
The interpretations of the value of those outcomes (and their causes).
The interpretations that can be made of those interpretations.

SOURCES OF INCREASED DIVERSITY IN NGO PROGRAMMES

At least six sources can be identified:

Growth in the numbers of beneficiaries involved in individual NGO programmes, usually associated with the growth in the budgets of NGOs. Proshika has seen the number of people it is working with increase from 162,000 in 1989 to 660,000 in 1994. The Bangladesh Rural Advancement Committee (BRAC) now has 1.5 million members but expects that this number will grow to 2.5 million by the year 2000.
Growth in the number of different locations in which members are located. Proshika now works in 70 different locations. Offices of the major NGOs in Bangladesh can now be found in most districts of the country.
Differentiation of services provided by NGOs. As NGOs grow in size there is a general, but not necessarily universal, tendency for the range of services they provide to increase. These may arise from internal reasons associated with individual pre-occupations as well as reactions to perceived donor or client interests.
The development of bilateral donor funding programmes for NGOs. The NGOs involved in such programmes create a second order of diversity, even within single sector programmes (such as those focusing on health and population). That diversity is in the various NGO representations of events at the beneficiary level, the activities, outcomes, and beneficiaries interpretations of the value and causes of those. This second order diversity is generated by the variations in the organisational structures, cultures and histories of the individual NGOs that are funded.
The development of participatory approaches to planning, implementation and evaluation. In principle, participatory planning should lead to the initiation of activities that are tailored specifically to local definitions of need and capacity. Participatory implementation should lead to variations in activities even where there are common plans across wider groups, again, because local circumstances, including accidental and chance events, will affect the shape of their activities. Participatory evaluations should also generate a multitude of interpretations, at least as many as there are members, though there are normally mechanisms for summarising those views.

This last source of diversity is more provisionally proposed. In practice, the participation of individual beneficiaries may be constrained by the aiding organisation, the surrounding culture and past experience. It may also be the case that there are some environmental events that have such a dramatic effect, such as large scale floods, that a uniformity of interpretations is generated, even with the free and unconstrained participation of all beneficiaries.

6. The unpredictability of events. While everyone may attempt to plan for the future the complex nature of reality always complicates that vision. Not only may necessary actions towards desired goals may become difficult or impossible to implement, but also judgements of what constitute good outcomes may be re-interpreted and perhaps even radically changed.

RESPONSES TO DIVERSITY

Three broad types of responses to diversity can be identified: (a) Ignore it, (b) Adapt to it, (c) Make use of it:

Ignoring diversity

The first approach essentially tries to ignore diversity. A foreign consultancy group has been working with Proshika to help develop an impact assessment and monitoring system. The former is meant to look at the longer term (4-5 years) impact of Proshika's programme, the latter, at the shorter term (1 year or so). Both are based on surveys of a stratified random sample of the 660,000 members, the first as members of households, the second as members of organised groups. The first will compare non-Proshika with Proshika households, the latter will compare identical groups over time different time periods. Since empowerment is the overall objective of Proshika, the consultant and the research unit have attempted to operationalise this concept by identifying indicators of empowerment and gathering data on changes in these indicators.

This approach is problematic for a number of reasons. Firstly, operationalising the concept of empowerment has predictably proved difficult. Through a process of consultation amongst Proshika staff the research unit has identified what they describe as 13 indicators of empower ment. In fact these turn out to be 13 sets of indicators, and the measurement of changes on these have required a total of 25 pages of questions. An immediate concern is the likely quality of the information given in response to questions on the 25th page. A second concern is that all of the questions seek information from within a frame of reference (assumptions of significance and causation) predefined by Proshika staff. Within the questionnaires no attempt is made to seek beneficiaries own evaluation of their experiences.

One donor did propose that the Proshika definition of empowerment should be validated by sampling a proportion of Proshika members and seeking their views. Predictably, this has not taken place. It is hard to imagine how a consensus could be established within 660,000 people, or even 10% sample, on a definition of empowerment. Even if this was achieved it would have to be at such a level of generality and/or ambiguity as to be impossible to operationalise with any reliability

Having gathered data from the sample survey Proshika was then faced with the problem of its analysis and interpretation. With the answers to 25 pages of questions from 1800 respondents which are to be analysed first ? Survey analysis typically takes a long time and timeliness of results is a factor affecting the value of survey research. Results were needed quickly for an immanent donor consortium meeting. Priority was in effect given to the analysis of some tables before others, leading to some data but not others being presented at the donor round-table. Who makes those choices ? Interpretations have to be made of those tables, and ideally they should be based on some contextual knowledge. Who makes those interpretations ?

Although the process of survey design and implementation appears very systematic and rigorous it is surrounded by human subjectivity. Firstly, what type of changes to focus on and what indicators might be indicative of them, and then later the interpretation of the meaning of the quantified results. In both processes there is a substantial diversity of options possible, and many choices made, which then leads to a focusing down on a limited amount of data and limited number of interpretations of that data.

The very process of quantification also involves loss of diversity. Counting requires an assumption of identity of the objects being questioned, one dowry less marriage was treated as the same as another, one person elected to the union parishad was treated the same as another. The process of summarising quantitative data involves further loss of diversity. Numbers are reduced to totals and averages. Rarely are measures of variance used. Any statistical analysis of differences is made on top of this process which has already reduced vast amounts of variation.

What is problematic here is not the fact of diversity being reduced, since it is hard to imagine any organisation dealing with 650,000 undiluted versions of reality every day. The problem is the unaccountability of the process, and the loss of information value that is involved in the process as it is conventionally structured.

Adapting to diversity

The second approach is what could be called a reformist approach. Some donors to Proshika would like to have seen the sample survey design of the impact assessment study complimented by a limited number of cases studies where PRA techniques were used to generate a richer sets of data that could shed some qualitative light on the data generated by the survey data. In some respects this reflects an awareness that the very process of "managing" diversity inherent in the large scale survey approach can lead to such de-contextualised data that it is difficult to actual make sense of the correlations found or their absence. This approach was not recommended for two reasons. Experience with a similar approach taken with a BRAC impact assessment study suggests that if the case study research results generate interpretations which are too contrary to accepted views they can be dismissed as having no validity because of the way they are sampled, they are after all, only case studies and their representativeness is not clear.

Another example of reformist approaches are attempts to take the process of indicator identification down to the people. In the late 1980 - early 1990's Action Aid was focusing on the use of a small globally applicable set of basic indicators of development. Later they abandoned this approach and focused on the local determination of appropriate indicators, project by project, but none of which were on the scale of Proshika's programme. The problem with this approach is that although decentralisation is one way of managing (as in adapting to) local diversity, diversity is then regenerated at the country or regional level in the form of a diverse population of indicators that then need to be made sense of collectively by more senior staff who have an ambit of responsibility that is wider than that of individual projects. How is that diversity to be managed ?

The case of Proshika also provides an example of a less conscious process of adaptation to diversity. In their Five Year Plan, for the period 1994/95 to 1998/9 there are separate objectives for each of the major sets of activities, those concerning the formation of grassroots organisations, education programmes, skills training, health related services, savings and credit, policy advocacy and others. These have all been consciously reduced down to one single overarching goal statement, referred to above, which is "The empowerment of all Proshika members". Less noticeable has been the process whereby the output indicators for these various activities have also been subject to a radical process of selection. In the Five Year Plan there are in excess of 30 pages documenting indicators of project outputs. In actual practice donors have funded a consultant to devise, in consultation with Proshika staff, a format for a Programme Status Report which will report each six months on the progress of the Five Year Plan in terms of outputs. Within the first of these reports there are graphs indicating performance on 24 indicators of programme outputs, in contrast to the 30 pages of outputs indicators specified in the Five Year Plan. In effect, Proshika's donors, and perhaps Proshika itself are monitoring a sample of output indicators. As above, the problem is not the fact of selection, an unarguable practical necessity, but the unaccountability of the process whereby it has taken place. There was no evident linkage from the contents of the Five Year Plan on to the contents of the ultimate summary, the Logical Framework.

Making use of diversity

The third approach is to try to make use of diversity. The methods proposed to Proshika mainly involve the use of qualitative data rather than quantitative data in contrast to both approaches outlined above.

One key organising principle suggested was that the voice of Proshika's members should be audible and distinguishable from that of Proshika staff, in documentation of either short or long term impact. Proshika annual reports, like almost all other NGO annual reports in Bangladesh, do not provide any space for the representation of members views. At the most there may be one or two illustrative case studies. In fact the structure of major documents such as annual reports works against this possibility. Sections are typically based on the different types of assistance provided by the NGO (e.g development education, income and employment generation, social forestry, universal education programme, popular theatre etc). The alternative, rarely seen, would be report sections organised around real groupings of people, and sub-sections organised around sub- sections of people, etc. Differentiations concerning the services they used and the results obtained would be sub-sets of these categories.

In this context of separate representation of members views it was argued that the results of the sample surveys should be seen as the voice of Proshika. They chose the methodology and they will choose the interpretations of the results. There is no need to throw out the idea of sample surveys altogether. What is needed are additional mechanisms to represent the views of the members.

Proshika's members are organised into 70 mass organisations. Each of these has four layers of representation, all but one of which are based on government defined administrative units: primary groups, the village, the union and the thana. All members of the primary groups make up the membership of village level groups. representatives from village level groups make up the membership of union level groups, representatives of union level groups then make up the one thana level group.

Each year all groups are involved an annual planning process, with the assistance of Proshika staff. Details of the plans are aggregated from the base level primary groups (PGs) upwards, with meetings starting with primary groups and moving later to the thana level. Although it was argued by some Proshika staff that such annual plans represent the views of members, in their final form of aggregated statistics of numbers of different types of activities, and respective budgets of expenditure, they are effectively unreadable as statements of opinion about the past years activities.

The suggestion was made that as part of this annual planning process qualitative information should also pass up the hierarchy of the peoples organisations and be consolidated as a statement of peoples views. Since there are up to 10,000 members per thana level groups a means of reducing the volume of information generated is necessary. Rather than summarising by inclusion, as is done when figures are added up, a process of summary by selection was suggested. The process could be structured as follows:

At the level of each PG, at the first stage of their annual planning exercise, the members would review the last years experience and from all the various activities they had been involved they would identify that which had been the most successful relative to others, in terms of their own criteria - what they thought was relevant. This would be documented on one page, a description of the activity and the reasons why it was seen as most successful (the criteria). The same would be done for what was seen as the least successful activity. (A 2 page report).
At the village level meeting members would review all the member PGs activities over the last year and identify which PG had been the most and least successful of all, relative to each other. Evidence would consist both of the reports from the PGs, (the documents create above) but also any other experience members could recall as relevant. Two pages would be created, one documenting the most successful village, the other the least, on the same basis as above. (A 2 page report, plus 2 x 2 page reports of their most and least successful PGs). 3. At the union level meetings members would review all their member village level groups and identify the most and least successful village groups. Again documentation brought from the village level would be of help, but not the only source. Again two more pages would be created, documenting the judgements made of the village groups. (A 2 pages report plus 2 x 2 pages on their MS&LS VCC, and 4 x 2 pages on their MS&LS PGs)

4. Finally, the same process would be repeated at the thana level, this time evaluating the union level groups. The same process of documentation would be completed. (A 2 page report plus 2 x 2 pages on their MS&LS UCCs, 4 x 2 pages on their MS&LS VCCs and 8 x 2 pages on their MS&LS PGs).

This process would result in a 28 page document summarising the judgements and performance of four levels of the peoples organisations. Two from the thana level, two from each of the most and least successful thanas, two from each of the most and least successful union groups selected by them, and so on, as shown below:

Thana....2

/..........\

/.................\

/.......................\

/..............................\

2..............Union...............2

/...........\.........................../........... \

/..................\...................../...................\

2....................... 2.Village..2..........................2

/............\............. / ....... \......./.........\............../.............\

2....................2.......2............2...2..............2 .....2....................2

Primary groups

The principle behind the documents structure is the sampling the edges of experience, rather than central tendencies. Successful activities, villages, and unions would be worth emulating, conversely unsuccessful ones need to be avoided. Both would be worth analysing in further detail. The definition of what constitutes success is no predefined nor is a single definition imposed. Although each group can use the criteria it sees fit, those criteria are made public and therefore held accountable. The definition of what each PG saw as success (event plus criteria) would be part of the evidence used in the judgements of what their superordinate village group saw as the most successful primary group. Not only would events be evaluated but criteria for evaluating them would also in effect be evaluated, as the process moved up the hierarchy.

Such a process of documenting a diversity of judgements of activities would still, in the case of Proshika, leave it with 70 (thana) x 28 pages of documentation. While 28 pages report at the thana level would be manageable, as a central office document either for internal use or for donors, a 70 x 28 pages document would be excessive. The suggested strategy for managing this involved a re-iteration of a similar process to the above using the hierarchy of Proshika staff who have supervisory/liaison roles in relation to the peoples organisations. One way of viewing the process of managing 70 x 28 pages is in terms of creating a map which would gives to some structure to the 70 reports. The metaphor of a map is significant. A map is different from a story in that different readers can start where they like, follow a route of their own choice and end where they like, according to their own preference and need. Nevertheless it contains verifiable information.

It was suggested that the echelon of staff immediately overseeing Proshika's relationship with the thana bodies (called Zonal Coordinators) would select and document which of the thana level bodies they liaise with was the most and least successful over the past year, according to their own judgement. Their bosses (Area Coordinators) in turn would select from these sets of choices what they thought were most and least successful, using their own criteria, but in the light of information and criteria suggested by Zonal Coordinators. Finally, the Coordinator of these staff would make a choice of the most and least successful thana group, from amongst these sets of choices.

As with the process at the level of peoples organisations, although many different criteria of success would be used, these criteria themselves would be subject to review by the next level in the hierarchy.

Problems with such an approach

1. Monitoring systems are instituted and used for a number of reasons, not only to generate meaningful data. The existence of monitoring systems established on orthodox lines is a symbol of capacity and competence, a potentiality which for many observers may be sufficient in itself. The methods outlined above are not yet seen to have any legitimacy. They are not well known and the experiments with them have been confined to one other organisation in Bangladesh and one in Malawi. Even if an NGO feels they may be of value their use might put at risk their own legitimacy as an organisation.

2. One particular concern of Proshika staff, and of some donors, was the explicit subjectivity involved, a dramatic contrast to the apparent objectivity of quantitative based impact assessments. Two points were made in reply: (a) Proshika staff in line positions responsible for services to members act daily upon their subjective view of the world and this has consequences for the Proshika programme. Proshika's policies are filtered through, and enacted on the basis of, their staffs interpretations; (b) The variability of peoples subjectivity is likely to be less of a problem, and more of a potential resource, if it is brought into the public domain and made subject to explanation, and choice by others.

3. Some staff and donors were concerned with the costs, mainly in terms of time, that such a system might impose on Proshika's members. There are two issues here, the actual time demands likely and the reasonableness of those demands. In respect to the specific proposal made it could be argued that the annual process of reviewing last years activities should already be taking place and what is involved is simply documenting that process. Two pages per group per year should not create an unnecessary burden. The process of documentation might actually generate a closer attention to the evaluation process, a benefit in itself. Regardless of this potential immediate benefit, there is unarguably a cost, in as a much as any members time is necessary. This does not need apologising for. Development aid is not free, the cost is information back about the beneficiaries and the benefits (or the lack of). Providing information takes time and time costs people either money or lost opportunities. Beneficiaries of aid projects usually seek to minimise these costs, and there is no reason for requiring NGOs to unnecessarily add to them. The question is whether the information generated by peoples participation in any monitoring or impact assessment system make sufficient difference to the quality of the assistance they receive. Rarely are monitoring systems evaluated in these terms.

4. Another concern that was raised by Proshika was how to relate the results of members own evaluations with those of the survey based impact studies - a problem arising from a diversity of methodologies. One is quantitative and the other qualitative, one tries to produce aggregate statements about all groups based on a small representative sample, the other focuses on a small biased sample of those groups identified through the participation of all groups.

One means of managing the difference is for Proshika staff to make their own choices of what constitute successful and unsuccessful primary groups, village groups, and union groups, based on their impact study data, and then for their choices to be compared to the choices made from the same set by the members during their own evaluation process. It was suggested that what is important, regardless of the data sources behind these choices, is the extent to which the two groups evaluations agree or disagree with each other. Agreement means joint action, either relating to success or failure groups, will be less problematic. Disagreement will mean joint action is likely to be problematic but at least that disagreement can be discussed and explored. However, ignorance of each others views (both of agreement and disagreement) is likely to be even more problematic, if there is any desire for joint action.

In these suggestions there are the seeds of two meta-indicators of programme progress. They are (a) the extent of agreement between parties on what constitutes success and failure, based on choice of real events, not just abstract descriptions (mission statements, project objectives etc); (b) the awareness of differences of view between parties on what constitutes success and failure. Emphasis on project impact in terms of changes at this level should fit well with a multi-stakeholder oriented view of development projects.

This leads back to another proposal for the management of diversity amongst Proshika's members, closely linked to the core idea of giving members the right to an independent voice of their own. It was suggested that if the members are empowered then they should feel able to express their differences of view with Proshika staff. In fact we should look for evidence of empowerment first of all in the relationship members have with their assisting NGO, Proshika. Here is where the results should be first observed, and most easily accessible. If this view of empowerment is acceptable Proshika (and it's members) should be able to identify each year, or any other period, what they considers as the main difference of view that arose in it's relationship with each of the peoples organisations. If it could not the prospects for the resolution or management of those differences would seem to be poor and the status of their empowerment would seem weak.

Alternate sources of judgement

The exploration of awareness of agreement and disagreement amongst different parties requires raw materials - information about judgements and views. Judgements require contrasts and choices. In the impact and monitoring studies described above two types of contrast have been made, between Proshika and non-Proshika households and between groups at different periods of time.

There are of course a range of other contrasts that could be explored as sources of judgements. The proposals outlined above introduce the use of comparisons of activities within the same period of time (at the PG level), and the comparison of different categories of Proshika groups (village and union) within the same time period. Other forms of comparison are also available. With the spread of NGOs across many areas of Bangladesh it is increasingly common to find more NGOs providing the same type of service in the same location. Hence it is now possible to ask members to compare the attributes of specific services provided by a number of different NGOs, as has been done by CCDB with their savings and credit programme. Members were quite able to say what aspects of each credit programme they thought was best and worst. What would have been of additional value, in addition, would be to have field level and other staff of organisations such as CCDB make their own judgements of the same service and then for these judgements to be compared to those of their members.

Other forms of analysis of diversity

The results of research studies, both survey and PRA based, provide another opportunity for the exploration of diversity, but with a focus on intra-NGO differences, rather than those between the NGO and its members. In the case of the Proshika impact assessment study a meeting was held in April 1995 between research unit staff and other senior staff to discuss and analyse the results. Staff were asked, in the case of specific variables such as immunisation rates, what their expectations of differences between Proshika and non-Proshika households would be, and their reasons for these expectations. The actual results were then provided and the differences between expectations and actual results discussed and analysed. This process of contrast in effect documents the impact of research, an outcome that is rarely systematically studied or of any explicit concern.

A higher level of diversity can be found through an exploration of the nature of the apparent agreement between two parties such as staff and members over the quality of services or performance of groups. In the case of CCDB an experiment of this kind was undertaken in their 1994 round-table meeting with donors. Five groups (beneficiaries, junior staff, senior staff and two groups of donor representatives) all examined an array of six changes (in the lives of specific beneficiaries) seen as most significant by CCDB staff within the context of a sample of 400 peoples organisations monitored over six months. The five groups choices of the most significant of these six changes, and their explanations for those choices, were reported back to each other in a plenary meeting at the round-table. A donor group and the senior staff group both agreed upon the choice of one change as most significant but differed substantially in their reasons, one focusing on the existence of harmonious relations between the men and women involved, the other more on the wider income generation effects. Although there was no explicit gender related observation category in the monitoring system the process of choice and comparison of choice that took place in the round-table meeting showed that important differences in the meaning of key events centred on gender issues.

Another form of internal diversity which is yet to be explored concerns what could be called organisational self-awareness. Staff members have views as to what groups and activities are the most and least successful within a specific period and location. They may also feel those views are shared by other staff members or in fact that they are largely disagreed with. Combining these possibilities together it is possible that in respect to a particular question of performance there may be four extreme situations: (a) a real consensus - everyone sees group x as successful (or unsuccessful) and understands that everyone else is of the same view; (b) a false consensus - few people see group x as successful but all believe that everyone else thinks it is; (c) a real dissensus - half the staff concerned see group x as successful and they understand that half of the staff disagree; (d) a false dissensus - all agree that group x is the most successful but they all believe that many people do not agree. While these may seem abstract possibilities the existence of a false consensus or a false dissensus could seriously affect the functioning of an organisation. The absence of either could be seen as an indicator of effective communications (informal and/or formal) within an organisation.

FINAL RESERVATIONS

The choice of how to manage diversity is not simply constrained by the availability of legitimised methodologies. One likely factor is the nature of external information demands. It seems that in many cases donors have not encouraged or helped created a demand for differentiated views of NGOs members beyond a few simplistic and globally applied categories (e.g economic class and gender). Perhaps the increasing emphasis on multi-stakeholder perspectives on projects may make some difference.

Gregory Bateson has defined information as "differences that make a difference". Information defined in this way is certainly closely related to power. People do not necessarily readily disclose information that gives others power or reduces their own. To the extent that they do may be indicative of another potential meta-indicator of programme progress, the establishment of trust in relationships between partners in development activities. This concept of trust is a more energetic one than the more common one based on passive acceptance of the intentions of others in the face of an absence of information.

Understandably, Proshika staff were reluctant to admit to any differences existing between themselves and their peoples organisations. Other organisations I have worked with, in evaluation contexts, have also been reluctant to admit to internal differences. Methods of monitoring that do seek to make use of diversity by opening up the world of participants interpretations will need to seek ways of addressing security needs that will reduce the risks of associated with more public exposure. Unless this is done we will remain stuck with projects as presentations and perfor mances.

-oo0oo--

Last revised 7 May 1996

This page has had visitors since May 19th 1997.