Social research tools for

CBOs/NGOs
 Table of Contents

I. Planning, monitoring, review, evaluation and impact assessment for good project/program
cycle management ........................................................................................................................... 3
1.1 Assessment and Planning ...................................................................................................... 3
1.2 Review and evaluation .......................................................................................................... 3
1.3 Impact Analysis .................................................................................................................... 4
II. Designing and Conducting an Research ................................................................................. 5
2.1 Stages and Steps .................................................................................................................... 5
2.1.1 Pre-data collection ......................................................................................................... 5
2.1.2 Data Collection .............................................................................................................. 7
2.1.3 Post-data collection ........................................................................................................ 8
III. Methods of Data Collection ............................................................................................... 12
3.1. Document Review/Content Analysis ................................................................................. 12
3.2 Surveys: Questionnaires...................................................................................................... 12
3.3 Focus Groups ...................................................................................................................... 17
3.4 Observation ......................................................................................................................... 18
3.5 Participatory Learning and Action (PLA) tools .................................................................. 20
IV Ethical considerations ............................................................................................................. 32

2
 I. PLANNING, MONITORING, REVIEW, EVALUATION AND IMPACT ASSESSMENT FOR GOOD
PROJECT/PROGRAM CYCLE MANAGEMENT

Needs assessment

Planning
Impact assessment

Review and Implementation

evaluation

Monitoring

1.1 Assessment and Planning

Assessment and Planning are the processes of identifying and understanding a problem and
planning a series of actions to deal with it. The end result is to have a clear and realistic plan of
activities designed to achieve a set of clear aims and objectives.

A needs assessment simply, identifies, defines, and assesses the needs of a particular
environment or situation. An assessment not only identifies the needs of a particularly
community. It also enables an organization to put together a picture of a community or
environment before an intervention or action is initiated. This provides a good basis for
comparison after the intervention is started or completed, as you can measure the changes in
needs. If a community needed X, Y and Z before an initiative and then after the initiative was
only in need of Z, then you can begin to determine how your effort played a role in supplying X
and Y, while it failed to completely satisfy Z. In this way, a needs assessment, followed by an
impact analysis, provide 2 powerful tools for project evaluation.

1.2 Review and evaluation

3
A review is the assessment at one point in time of the progress of a project or program. The
basic purpose of a review is to take a closer look at a project or program than is possible
through the process of monitoring. Reviews can be carried out to look at any aspect of a project
or program, and can use a range of criteria to measure progress. Appropriate management
decisions can then be taken. Regular review can be conducted as part of a monitoring system. A
review can be formal or informal, broad or in-depth, and can be carried out internally or by
outsiders.

An evaluation is the assessment at one point in time of the impact of a proejct or program and
the extent to which stated objectives have been achieved.

Planning, monitoring, review and evaluation focus on the processes and direct results of a
project or program. Impact analysis focuses on longer-term and wider-ranging changes beyond
the immediate results of the work.

1.3 Impact Analysis

An impact analysis is another type of evaluation that measures the effect (impact) that a
certain action has on an environment. After implementing a program, whether short-or long-
term, you should be interested in knowing whether or not the program had an effect on the
target audience or community. Was the desired impact achieved? Why or why not? Impact
analysis is often a focal point of NGO evaluation efforts, as they reflect upon an NGO’s
performance, and illustrate to the donors that change is (or is not) occurring. An impact analysis
is an outcome oriented evaluation, in which the end-result is the primary interest.

At the most basic level, in order to accurately measure impact, it is important to know 3 things:

 What is the situation before the intended effort?

 What is the anticipated goal, effect or outcome?
 What is the situation after the action?

4
 II. DESIGNING AND CONDUCTING AN RESEARCH

2.1 Stages and Steps

There are 5 basic steps in designing and conducting a research:

Pre-data collection 1. Determine what you want to know and the goal of the
research
2. Determine the questions you must ask and the data you
must gather
Data collection 3. Gather data
Post-data collection 4. Analyze data
5. Compile and present your results

Roughly the steps can be divided into 3 phases: pre-data collection, data collection, and post-
data collection.

2.1.1 Pre-data collection

1. Determine what you want to know and the goal of the research

If you have already decided to undertake a research, you probably have a general idea of your
interests and expectations. However, you must clearly and specifically determine and define
what it is you want to evaluate, and what the ultimate goal of the evaluation is. What do you
want to know, and what do you intend to do with the results?

The following Worksheet provides an example of the basic question you should consider in this
initial stage of your research.

5
Worksheet: Research Purpose and Goals
Worksheet: Purpose and Goals of the Research

What are you going to identify and what do you want to know?

 Specific activity (note name)____________________________

 Specific target group/s (if any) (note name)__________________________
 Overall organization

Key Points of Contact:

1.
2.
3.

Is this research a (check all that apply):

 Needs assessment
 A mid-term progress evaluation
 An end-of-effort evaluation (for a single project or a larger project)
 An impact analysis
 A process evaluation

State the reasons that you are conducting this research:

 Donor/funder requirement
 Ensure quality assurance in an ongoing project
 Standard internal project management procedure
 Necessary needs assessment for proposed project

Who are the stakeholders of the effort? Whose interests are served?

What is the goal of the research?

Who will read the final report?

What is the timeline for completing the research?

6
2.1.2 Data Collection

Once you have determined the kind of information you need to conduct the research, begin to
think about how you will gather this data.

As a first step, think about the questions you want to ask and begin to identify the information
that you can gather through document review. Make a list of resources you already have,
resources you know about and need to obtain, and potential topics for future research and
identification. Depending on your research topic and goals, it might be possible to conduct all of
your research through document review. However, it is more likely that you will opt for a mixed
method approach, and begin to collect data from various target groups.

If you do want to gather feedback from various target groups, consider your questions, your
target respondents, and your collection technique options, and begin to determine which
techniques would be most appropriate based on the questions and the intended respondent.
When considering the appropriateness of various techniques, think both about your resources
and the interest and capabilities of the respondent. You may not have the resources to
personally interview all of the people you would like to survey, or to organize a focus group in a
limited amount of time. While questionnaires are good methods for many people, children may
not be capable of filling them out accurately, and often elderly people have similar difficulties,
or lack the experience in providing such responses. Pick methods that will have the highest
prospective rate of return, so that you gather the most data you can.

Data collection is possibly the most important part of a research, as you must have reliable and
concrete information available to you in order to assess needs, measure progress, or determine
cause and effect.

Bringing information together from a variety of sources is a good way to confirm or validate
your data. This process is called data triangulation, and it can help to ensure the validity of
your conclusions. If data gathered through your interview process yields similar feedback as
data gathered through a questionnaire distribution or through basic research and document
review, then you can have more confidence in the results. If the data you gather through one
method is very different than data gathered through another, then you should analyze your
results with caution, possibly even obtaining additional data to try to determine the reasons for
the discrepancy.

Qualitative Data and Quantitative Data

Qualitative data is data that is based on words. One criticism of qualitative data is that words
can be subjective and open to interpretation. However, a strength of qualitative data is that is
offers a contextual richness that numbers alone cannot.

7
Quantitative data is based on numbers. Quantitative data is often gathered through surveys. A
strength of quantitative data is its objectivity – it is raw information. However, a weakness is
that these numbers can be limited and limiting, are often dry and colorless, and fail to paint a
rich picture of a situation. They can also be manipulated.

A mix of qualitative and quantitative data can help to provide both the objective scientific basis
of your research, as well as the contextual framework needed to truly understand the
environment in question.

2.1.3 Post-data collection

Data Analysis

Once you have gathered your data, you must turn the raw facts and information into usable,
actionable analysis, praise, criticisms, observations, and recommendations. This section will
introduce some basic methods for data analysis, breaking analysis down into qualitative and
quantitative data analysis methods.

Organize the Data

Throughout your data collection effort you should organize the incoming information so that it
is manageable when you begin your analysis. It is worth noting that when you begin to receive
and organize data, though questionnaires, interview summaries, observation collection forms
etc., it can also be useful to begin to start to look at the responses. You may find that
respondents are misinterpreting a question, or are consistently making mistakes in filling out a
questionnaire. If you find such a trend early in the data collection process, it might be possible
to fix the mistake so that future respondents don’t experience the same confusion. Again, pre-
testing evaluations to gauge understanding in advance can help to minimize this problem.

You should code all of the feedback that you receive for easy identification. For example,
interviews might be coded I1, I2, I3, etc., with the label put in the upper right-hand corner of
each interview summary sheet. Questionnaires might be labeled Q1/1, Q1/2, etc.
(Questionnaire Version 1, Respondent 1, Questionnaire Version 1, Respondent 2). Coding your
data will make organization, analysis and feedback citations manageable.

Trends

Whether working with qualitative or quantitative data, the most basic part of your data analysis
will consist in identifying trends and patterns. Consider the following examples of potential
trends that an NGO might identify through its data analysis:

8
While reviewing feedback from women concerning their long-term professional goals, an NGO
noted a trend that indicated that respondents from cities more clearly articulated their goals
than women from small towns.

A questionnaire revealed that children who played video games for more than 2 hours a day had
lower grades in reading, spelling, and history.

A focus group indicated that while young people liked the new brand of candy, older people
thought it was too sweet.

The study showed that there was no difference in music tastes among children from poor and
wealthy families.

You may be able to clearly and immediately identify some trends, though others may be more
difficult to identify, and may necessitate detailed review and reflection.

Working With Qualitative Data

Your qualitative data will likely come from a mix of interviews, responses on questionnaires to
open-ended questions, document review, observations, and focus groups. For each question
asked, begin to compile the responses, so that all comments on Question 1 are noted on one
page, all comments on Question 2 are noted on another page, etc. Next to each response, note
the code of that specific form (Q1/4, I3. etc.), so you can find it for reference in the future.
Compiling information in this thematic way will enable you to spot trends visually more easily
than if you have to flip through the responses on 50 different forms.

Working With Quantitative Data

Your quantitative data will also come from a mix of sources, but will probably be heavily based
on information gathered through questionnaires and interviews. There are sophisticated
statistical analysis tools that can help professional to manipulate and analyze quantitative data.
Programs such as SPSS or other software tools are very useful if you have a large amount of
quantitative data, possibly gathered through mass distribution of a survey. However, you can
analyze quantitative data using a calculator to answer some basic questions. The most common
questions include the following:

 What was the average response? (For instance, “On average respondents rated the food
quality at a restaurant as “good.””)
 What percentage of respondents selected any one of a set of given choices? (For example,
“45% responded “Yes,” 35% responded “no,” and 20% responded “Undecided.””)
 Is there a difference in response trends among different respondent groups? (For example,
“Do respondents from Shan and Karen on average respond differently to questions
regarding interfaith marriages?”)

9
These simple questions can be easily calculated and will point out some very interesting trends
in your data.

Interpretation and Analysis

Once you have some basic understanding of trends and response frequencies, you can begin to
interpret and analyze this information. What does the trend mean? What does the data
suggest? Consider the following examples:

The data indicates that 80% of the children who completed the World Religions course module
had a favorable opinion of people from different ethnic groups, while only 23% of the child
respondents who did not complete the course noted a similar favorable opinion. As the children
were all from the same school, the results suggest that the course increases children’s respect
for diversity.

91% of the respondents noted that they would not feel personally affected by the elimination of
the library’s video-loan program, suggesting that the program could be cut without any serious
repercussions in the community.

Be careful to consider other explanations for cause and effect. For instance, in the first example
above, the analyst might have to note the following:

However, it should be noted that because parents made a choice on whether or not their
children should participate in the course, it is possible that the more tolerant attitudes were also
fostered through the children’s home environment, and not just through the course.

Review the results gathered through data collection effort, and pay attention to how they
organize the data, and how they identify trends. Later, these trends will be turned into a
summary of general findings and recommendations.

Evaluation Report Development and Presentation

After you have done the difficult work of designing a research strategy, collecting data, and
conducting data analysis, the preparation of the research report is relatively simple. You have
all of the component pieces, and now need to put them together into a cohesive and concise
whole. Whether you are preparing the research for internal use only, or for external
distribution to a donor, you may decide to emphasize different results or findings. However, in
general your report should include the following 5 elements:

1. Executive Summary: Provide a short (2-3 paragraph) summary of the main issues
evaluated, the main findings of the research, and recommendations
2. Introduction: Explain the issue being conducted research and the reason for and goals of
the research (needs assessment, impact analysis, process evaluation, or a combination of
the 3)

10
3. Methodology: Explain how you developed your questions, your data sources (respondents),
and how you gathered your data
4. Findings: Explain the results of your data analysis
5. Recommendations: Explain your recommendations based on the results of the research

This information can be compiled in a report and distributed to your staff, if for internal
purposes only, or more broadly distribute, if the report is for public consumption. You may
want to (or be required to) prepare a short presentation for your donors in which you orally
summarize the main results and recommendations, possibly using a PowerPoint slide
presentation to draw attention to key results.

11
 III. METHODS OF DATA COLLECTION

3.1. Document Review/Content Analysis

Document review, or put more simply – reading – is always a necessary part of any research
project. It is important to identify the work that has already been done on a topic, and the
information that has already been gathered, not only so you don’t recreate the wheel, but so
you become aware and familiar with the issues and perspectives on the topic.

When you begin a data collection effort, start by finding out what reports, books, and journal
articles have been written on the topic. You may find that other NGOs have already conducted
similar research, and you can learn not only from their results, but from their process.
Government offices might have statistical data available.

One you have gathered the information, begin to critically review its content, preparing
summaries of highlights that can contribute to your own work. Be sure to cite any sources you
use, to avoid plagiarism. Analyze the information, and determine how it can fit into your own
framework for evaluation.

Pros and Cons

Document review can help you to become more familiar with a topic outside your immediate
realm of understanding. Libraries and the Internet provide a cheap (often free) wealth of
information that can contribute to your data collection effort. While work may have been done
by others, you can build on this work by adding your own data, analysis, and conclusions, and
therefore contribute to the body of knowledge available in the literature.

There are really no weaknesses to reading, though the researcher should avoid some potential
pitfalls. First, be aware of the source of information, so you can measure its validity or bias.
Second, review all of the documents you have collected, and be prepared to synthesize all
relevant information. Don’t just decide to leave out information that does not fit your preferred
conclusion; if you don’t think it is accurate, include it and explain why. Third, access to quality
documents can be difficult, either due to limited library facilities, or a reluctance on the part of
other organization or agencies to provide data and reports.

3.2 Surveys: Questionnaires

Surveys and questionnaires are methods of collecting similar data in an organized way from a
potentially large population. Strictly speaking, both questionnaire and interviews are methods
of collecting data through surveys.

Questionnaires are written forms that ask the respondent to answer a series of open- or close
ended questions. Questionnaires can be printed or can be conducted on the Internet. There are
several different ways a data collector can distribute a questionnaire to a population:

12
 Personal delivery; the researcher gives the questionnaire to the respondent
 Mail delivery: The questionnaire is mailed to the target group, and respondents are asked to
return the filled out forms
 Respondent self-pick-up: A pile of questionnaires can be left for respondent pick-up, either
in your office, or in a community center that attracts a lot of traffic. These can be as
complex as a multi-page form, or as simple as a customer service response postcard left on
a table in a restaurant or hotel.

It can be very useful to test a draft questionnaire on a small number of people, to ensure that
your instructions are clear and that respondents can easily and accurately provide the
information you request.

Pros and Cons

One strength of questionnaires as a method of data collection is that they enable the potential
collection of a large amount of qualitative and quantitative information. Another strength is
that questionnaires can be anonymous or confidential1, giving respondents a safe way to
express their true concerns or opinions with out fear of a potential backlash.

A weakness of questionnaires is that they are very time-consuming. Your staff must develop the
questionnaire, distribute, collect, and analyze the data. Additionally, response rates can be as
low as 10%, meaning that in order to receive 20 fully completed questionnaires, an NGO may
have to send out 200 blank questionnaires to potential respondents. To ensure maximum
return, you may need to continually call respondents and ask or remind them to fill out your
survey. However, as people are bombarded with more and more questionnaires, even this
“personal touch” may not be enough.

Surveys: Interviews

Interviews are another method of collecting data using a survey technique. They are similar to a
questionnaire, but with one key difference: they are conducted directly between the researcher
and the respondent. Interviews are traditionally done in a personal, face-to-face meeting.
However, they could also be conducted over the telephone, or even through a question and
answer email discussion. Interviews tend to be one-on-one, but can also be held with a small
group of people.

Like questionnaires, questions can be open-ended or close-ended, with the goal of obtaining
qualitative or quantitative data, though interviews are a very good method of obtaining
qualitative information through open-ended discussions. Whereas questionnaires can be
limiting in their potential for qualitative data collection, as the respondent must write out a

1
An anonymous source has no name or identification associated with the response. A confidential source may agree to provide
their name, but with the understanding that their identification will not be revealed.

13
response, may run out of room and be uninterested in attaching additional sheets, or may not
want to spend their time writing, an interview can evolve into a sort of conversation.

Keep the following basic tips in mind when conducting an interview:

 Make an appointment for the interview. People often do not have time for unscheduled
visitors, and you must demonstrate that you respect the person’s time.
 Prepare a set of questions to ask the respondent, possibly based on a questionnaire–like
form to ensure consistency and to ensure that you do not forget an important question.
 Take notes during the interview, not only for your benefit, but to show the interviewer that
you are interested in what they have to say. Type up an interview summary soon after the
meeting, so that you can capture information as accurately and completely as possible.
 Make eye contact, and demonstrate positive and engaged body language.
 Send a thank you note after the interview, to show your appreciation for their time.

You may also be interested in recording the interview on an audio cassette to facilitate full
transcription of the interview, or future reference. If you decide on this method, be certain to
inform the respondent, as some respondents may not agree to being recorded.

Pros and Cons

A strength of an interview is that it enables a researcher to gather information directly from the
source, whether from a beneficiary, an expert, or an active community member. The
interviewer can get more detailed responses than they can solicit through a questionnaire, as
people often talk more freely than they write. The interviewer can also feel free to ask relevant
follow-up questions, and take advantage of an interviewee’s apparent willingness to share their
experiences and opinions. Finally, an interview provides another way for a researcher to put a
public face on their project, and to meet the people involved or affected by an NGO face to
face.

A con of interviews as a data collection technique is that they are time consuming – not only for
the researcher, but for the respondent who has agreed to take time out of their schedule to
meet. Because they are time consuming, they can also be expensive if your staff must take
time out of their schedules to conduct the interviews. Additionally, you may find that while
some people will agree to be interviewed, they are unable or unwilling to provide you with the
information you had expected. In this way, you may find yourself having conversations that are
not as relevant as you expected.

Asking Questions

Basics Do’s and Don’tsBe consistent: When asking questions, be sure to ask questions in a
consistent manner to ensure standardized response. Consistency of data collected can help to
ensure consistency of data analysis.

14
Don’t lead the respondent: When developing your questions, be careful to phrase them in a
way that is neutral and objective. Avoid phrasing that immediately implies a judgement on your
behalf, or that forces a certain tone from the respondent.

Be thorough but random: Try to seek responses from as many people as possible to ensure that
your data set is thorough. However, also ensure that you select your target respondents so that
they represent a broad segment of the population. Don’t just interview your friends, and don’t
just ask people who you know will provide you with a favorable response, as such practices are
unethical and will skew results.

Open- and Close-ended Questions

Questions can be either open-ended, close-ended or a combination of both. An open-ended

question allows a respondent maximum freedom and flexibility in providing their views or
opinions. The following are examples of open-ended questions:

Questions that can be asked in a needs assessment and planning

There are many different types of questions that can be asked when conducting a needs
assessment, and the specific questions will vary according to the issue under consideration.
Very simply, there are 2 questions that need to be asked and answered:

 What is available?
 What is not available?

Information Source of Means of Collecting Comments

Required Information Information

Determine the questions you must ask and the data you must gather

Next, think about the people you need to talk to or contact in order to begin to answer these
questions. It is possible that all of your data could come from secondary sources, but it is likely
that you will need to gather feedback directly from the people involved in your project.

15
Your target respondents could include a variety of populations that have direct or indirect
contact with your initiatives on the ground, including:

 Direct beneficiaries of an initiative

 Community members
 Local officials
 Implementing partners

When considering your potential target respondents, think about the potential impact your
programs have on the broader community, not just on the users. For instance, if you are
interested in finding out if a diversity awareness workshop improves participants’ perceptions
of the community, you should ask the workshop participants, but also ask the target group
themselves. The participants may perceive that their views have changed, but the target group
may indicate that nothing has changed at all. Gathering data and feedback from a variety of
sources ifs critical to uncover many different views.

There are really 2 sets of questions that you must be prepared to ask. The first are the
questions you will be asking yourself, in other words the questions that you want answers to.
The second are the questions that you will eventually actually pose to your target audience.
These will be very similar, but will have slight differences in presentation. Consider the
examples in the below box.

Asking Questions
What the NGO wants to know…. The questions asked of the respondents…
Do our gender awareness activities have an impact Did our gender awareness activities have an impact
on the way people behave? on the way you treated men/women in the
community?
Did our computer training program make it easier Did participation in our computer training program
for people to find jobs? make it easier for you to find a job?
Do teachers notice a difference in the behavior of Do you notice a difference in the behavior of
children who have participated in a conflict students who have participated in a conflict
resolution skills training shop in comparison with resolution skills training shop in comparison with
students who have note? other students who have not?
When getting started, ask your research team to brainstorm as many potential questions as
possible, to generate ideas and promote fresh thinking. Write down every idea the group
suggests, to encourage free thinking and creativity.

Next, begin to group these options into thematic categories. You may discover that many of the
questions proposed are very similar in content and goal, and just different in wording.

You may want to ask a million questions. However, you need to limit your questions to a
manageable number (people won’t want to fill out a 20 page survey!). Think about the
questions that will provide the most revealing and important information, and include them.

16
Discuss each proposed question and ask your research team why that question is important,
and what it can reveal. You may find there are cases where a very interesting question is
proposed, but on second look find that it is not really relevant to the evaluation goals. Don’t
worry about wording, phrasing or organization, as you will be able to address those concerns
when you begin to select your data collection technique options.

The Sounds of Silence

When interviewing a subject, learn to be comfortable with silence. If you ask a question and
receive a short answer, do not rush to fill silence or gaps in the conversation. Respondents
often need time to formulate their own responses, and if you cut them off, you won’t obtain all
of the information possible. Additionally, respondents will often feel uncomfortable with a gap,
and begin to volunteer more information than they originally intended. Use common sense to
determine when a gap is long enough, and when it is the appropriate time to move on to the
next question.

3.3 Focus Groups

Focus groups are small groups of people that are brought together by a researcher to discuss a
certain topic. In some ways, it is similar to having an interview with many people at the same
time. Focus groups can be used for many different kinds of evaluations, but they are very
common in consumer product testing and marketing, as companies want to ensure that their
product is appealing to as wide a group as possible. In the NGO community, focus groups can
be a lot like workshops or community forums, in which people are brought together to air their
opinions on a variety of topics.

Consider the following potential examples of focus groups:

A group of 15 teenagers takes part in a focus group in which they listen to potential up-and-
coming musicians, to give the CD producers an idea of what styles and varieties of music they
prefer.

10 women are brought together to try a new series of beauty products, both to fine-tune
production and packaging, as well as to provide data for an effective marketing campaign.

A group of youth are brought together to talk about what frustrates them about their
classmates and teachers in school, to help a group of educators develop more effective
integration methods.

Focus groups tend to have an average of 8-15 people; enough people to ensure a broad range
of perspectives but not so many as to make the event unmanageable. Focus groups generally
shouldn’t last more than 1-2 hours, as people have busy schedules and your NGO should be
prepared to facilitate efficiently. Formulate your questions and interests in advance so that the
focus group can stay focused and relevant.

17
Be sure that the setting for your focus group is pleasant, that the room is comfortable, and that
you have light refreshments. Have supplies on hand (flip charts, markers, etc.) if you want the
focus group members to be actively involved. Also, be sure to have adequate NGO staff on
hand so that they can facilitate, observe, and take notes on the proceedings. You may want to
integrate another data collection technique into the focus group by passing out a
questionnaire. This provides another standard way for participants to express their opinions,
and will provide you with a “captive audience.”

After a focus group, you should begin to compile the information from the observers and
facilitators as quickly as possible, to ensure that lapsed time does not dull your impressions or
memories.

Pros and Cons

A strength of focus groups is that they enable you to get feedback directly from your target
audience. Just as one interview enables you to spend time with an individual in depth, a focus
group with many people allows you to begin to develop more personal contacts with your
users, and to more broadly understand their needs. Additionally, a group of people may
provide difference feedback than they would as individual respondents, as an interactive
conversation may lead some people to think more broadly and creatively.

A weakness of focus groups is that they are resource intensive, as they take time to organize,
plan, and conduct. In this sense they are also expensive, as you must factor in not only the
costs of the focus group itself (meeting place, supplies, refreshments, etc.), but the time spent
by your staff planning it. Another weakness is that participants may have their own agenda in
agreeing to come to the focus group. One or two aggressive participants may dominate the
event by venting their personal opinions, thereby potentially skewing the results. Effective
facilitation an help to minimize this possibility.

3.4 Observation

Observation is another data collection technique that enables you to reflect on your own
observations and impressions of the way an activity or project has filled a need and made an
impact, or how a process was executed. You may have observed first-hand the growing
friendships of children from different ethnic groups who met on a team-building fieldtrip. You
may have seen a woman’s self-confidence grow as she learned professional skills. These are
valid forms of data that need to be captured to create a complete picture. Your experience as a
first-hand participant or participant-observer is important, as you can bring expert knowledge
and insider perspective to an evaluation.

Capture the observations and comments of people on your team by asking them to share their
anecdotes and stories. People you survey may share observations of their own. Collect as many
observations as you can, putting them on paper so that they can be included in the final data
analysis.

18
The main strengths of observations is that they are first-hand, personal sources of data.
Changes and impact may be easily seen in a certain situation, and such observations are often
valid. A weakness of observations is that people may see what they want to see, even if the real
situation does not correspond with reality. Observations can also be subjective, as 2 people
observing the same event may have very different interpretations and recollections. However,
these factors should not discount their use, as they can complement more formally compiled
data and add richness and color.

19
3.5 Participatory Learning and Action (PLA) tools

3.5.1 Resource Map (Source: adapted from Callens et al. and Berg et al.)

Description

A Resource Map is a map showing a community’s or group’s living environment, natural

resources, social facilities and infrastructure

Objectives

· to gain an overview on the situation in the area as perceived by the people living there

· to find out and discuss different perceptions of various groups on problems connected to their
living environment

· to learn about access and control over resources

· to discuss potentials, problems, needs, and visions arising from the analysis and discussion
processes while drawing the map

Key Questions

- which resources exist, which ones are rare or missing

- where are resources and social facilities located

- who can use which resources

- which resources, facilities, places, infrastructural items are known and important for the group

- which are the main problems and potentials as perceived by different interest groups in the
area

Remarks/Hints

The Resource Map is a good tool to with which to begin with. It is easy and fun for local people
to do. It helps initiate discussion amongst the group members themselves and with the
PRA/PLA-team. Apart from starting a joint process of information gathering and analysis, the
Resource Mapping is also a helpful tool to make outsiders familiar with the people and their
area, because it provides an overall orientation to the features of the group and its resources.

In most cases, it will make a lot of sense, to go through this map with separate sub-groups, e.g.
men and women in the village. This is because women and men often do different work and

20
use different resources, which results in a different focus on what are important features.
Resource Maps may include: water sources, firewood sources, grazing land, infrastructure (e.g.
roads, buildings, bridges), shops, markets, health clinics, schools, mosques, churches, special
places (sacred sites, cemeteries, shrines, etc)

Facilitation

1. find a large place, if possible with soft and sandy ground easy to draw on

2. start by placing a rock or leaf to represent a central and important landmark

3. ask participants to draw a map of the area they are living in, showing all important items
(according to their perception), such as natural resources, buildings, infrastructure etc.

4. encourage them to use symbols or natural materials to visualise the different items

5. take care, that once somebody has made a statement, you ask the others, whether they
agree, disagree or want to add something - encourage again and again the participation of all
persons present

6. Once they stop, you can ask whether there is anything else of importance that should be
added

7. when people stop working on the map, ask them first, if all participants feel, that the map is
complete and shows all important things in their living environment

8. then start asking questions like: "which of the things we can see on this map are helpful for
you" - "which ones are bothering you" - "to whom belongs this" - "who decides how to use this
and that resource" - "where are the problems in your area" etc.

9. after having discussed the characteristics of the area and the actual conditions of living there,
ask people to show you, how they would like it to look like after 5 years, what could be donel to
reach such a situation, who would be the persons/groups to decide upon and enhance such
changes.

10. make sure to note down carefully all important points of discussion and any other
additional information.

11. don’t forget to draw a copy of the map before leaving - insert the legend and cardinal points
(ie.North, South, East, West).

12. before closing the session, thank participants for their contributions

21
13. it can be very helpful, if facilitator and note-taker sit together after the session and compare
their observations in order to complete the notes

3.5.2 Mobility Map (Source: adapted from Berg et al.)

Description

A mobility map is a map which shows patterns of spatial mobility for different groups within an
area/community subgroups.

Objectives

· to find out about patterns of spatial mobility for different groups and sub-groups within an
area (e.g. women/men, elders/youth, herders/farmers etc.)

· to analyse contacts of group members to persons outside their group

Relationship Diagram (Source: adapted from Berg et al.)

Description

A Relationship Diagram shows institutions, organisations, groups, individuals, as well as their

importance and co-operationality as perceived by for the people.

Objectives

· to identify institutions and organisations which are important for local people

· to identify persons/groups, which are important for decisions/actions amongst local people

· to show the level of co-operation between institutions/organisations/opinion

leaders/decision-makers for the local people

· to find out and discuss differences in the perception of such institutions/organisations/groups

amongst local people

Key Questions

- which institutions/organisations do local people know? What impression do local people have
of these institutions?

- how good is the contact/co-operation between these institutions and local people?

- how important/effective is the support of such institutions for the local people?

22
- how do the local people see existing groups/important persons amongst themselves what are
the contacts/relationships like between different groups amongst local people?

- how strong do these groups/persons influence local decision making?

Remarks/Hints

If you have a feeling that it is difficult for the people to understand this tool, give a simple
example by drawing e.g. a Relationship Diagram showing a family and institutions important for
them, such as e.g. school, hospital, working place, nursery, shops, sport club etc.

Facilitation

1. choose the material you want to use (drawing paper or sandy ground)

2. question and discuss with the people, what is their individual view on all persons, groups,
organisations and institutions which affect their lives/working context, this can be e.g.
traditional leaders, government agencies, co-operatives, projects, development organisations
etc. - invite them to write a list

3. let them draw a circle in the centre of the paper or on the ground representing themselves

4. then each institution, organisation, group, person shall be represented by a circle

5. ask them to draw smaller circles for those which are less important and bigger ones for those
which are more important

6. ask them to indicate the degree of contact/co-operation between them and the institution in
the form of the distance between the circles representing them:

well-separated circles - no/little contact or co-operation

circles close to each other - loose contact or co-operation

touching circles - some co-operation

overlapping circles - very effective co-operation

7. ensure that the placing of circles and their sizes is a result of consensus within the group

8. mark the circles with names and/or symbols

23
3.5.3 Matrix Scoring and Ranking (Source: adapted from Berg et al.)

The prioritisation and decision-making processes about various subjects and - problems,
potentials, needs, wishes, constraints, solutions, project ideas - are an integral part of most
participatory processes. Training participants have to become familiar with ways of facilitating
such exercises.

Simple Ranking

Description

Simple Ranking (voting) serves to facilitate priority setting and decision making. It can be used
with large groups of people and is less time-consuming than other ranking and scoring tools.
Objectives

· to support a group of people in coming to a decision on their priorities concerning a

problem/need/project idea/vision

· to compare and discuss preferences of different groups

Key Questions

- which item (e.g. a certain problem, project idea) out of several ones is looked upon as most
important, favourable, necessary or pressing by a certain group of people

- how different are the preferences between interest groups amongst local people
Remarks/Hints

· as there is the danger that persons can be influenced by others or by the votes already cast
(there seems to be a tendency to vote for those items which have already a majority of votes),
you may decide on secret ballots

· one problem about Simple Ranking is, that there is no transparency about the criteria for the
decision, so that people might not be content with the results and/or change their minds when
they start discussing it afterwards - in such a case, a Matrix Scoring is to be preferred
Facilitation

1. ask the people to list the items to be ranked by writing them down and adding symbols

2. ask them to take little stones or seeds (the same number for each person) and to allocate
them to the different items - a higher number of stones for the item they prefer most and
fewer or no stones for those which are not so important to them

3. when everybody has finished voting, count how many votes have been given to each item

24
4. list the items according to their ranking and discuss with participants whether the results
represent their opinions

3.5.4 Matrix Scoring

Description

Matrix Scoring is a tool by which certain issues are ranked against a list of criteria agreed upon
by consensus within a group.

Objectives

· to rank and prioritise identified items (e.g. values of various shrubs and plants), objects,
problems, solutions, project ideas

· to make the reasons and underlying criteria for this prioritisation or decision clear

· explore the trade-offs made during the process of choosing

· highlight the criteria that are high (best) and low (worst)

· explore the differences between different groups’ perception

Key Questions

- which of the items under discussion are prioritised by different groups of the local people?

- which criteria are used by local people to determine their priorities

- how do people rank these criteria?

- how different are the preferences and perceptions between different groups (e.g. men -
women) within the local context?

Remarks/Hints

· be aware that Matrix Scoring is one of the more difficult tools, but perfect for producing
decisions based on clear criteria

· an appropriate size of the matrix is: not more than 5-8 items and not more than 5-8 criteria

· be aware, that a simple total of the numbers of each column in the matrix can give misleading
information, as it implies an equal weighting for each criterion

25
· make clear that the process of discussion and debate that occurs as the options and criteria
are selected, as well as a consensus of the group about the final decision is more important
than the simple total of final scores

Facilitation

1. explain to the group, that this tool helps to prioritise things in a transparent way, discussing
openly the reasons why some of them are perceived as being more important than others

2. invite participants to draw up a matrix with the items (e.g. plants and shrubs) to be
prioritised along the top row

3. in order to identify the criteria against which you are going to rank, ask them, what is
important for them concerning the respective items, or -what is good/bad about one or the
other

4. list the criteria thus identified down the left column and invite people to search for symbols
representing them

5. make sure that all criteria you are listing, are (according to the options to be prioritised)
formulated from the same perspective - either positive or negative, otherwise, the quotes will
reflect an incorrect basic assumption.

6. then ask people to place their scorings (dots, stones, seeds) on the matrix

7. afterwards ask them to count the totals per option and analyse the result of the scoring

8. lastly, encourage a discussion about whether everybody feels that the result represents the
groups opinion - otherwise invite them to decide upon which criteria are most relevant to them
and make a second scoring using only these new criteria and compare them with the first result

3.5.5 Pairwise Ranking (Source: adapted from PL&A and Berg et al.)

Description

Pairwise Ranking or Preference Ranking is a tool to set priorities between different options
available. In Pairwise Ranking, each individual item is compared directly against the others so as
to emerge with a ranking from highest (best) to lowest (least).

Objectives

· to determine the main preferences/priorities/needs/constraints or problems of individuals or

groups for a set of items/subjects/topics

26
· to compare the priorities of different groups against one another (e.g. men - women, young -
old, rich - poor etc.)

· to support a group of people to discuss, analyse and prioritise their problems, needs and
possible solutions

· to find out about criteria for decision-making

Key Questions

- which item out of several ones is looked upon as most

important/favourable/necessary/pressing bay a certain group

- which are the criteria for preferring one item to another

- how different are the preferences between different groups within the local population
Facilitation

1. draw a matrix - indicate the list of items you want to compare across the top by using
symbols and the same list within the same sequence down the left column, so that each open
box or cell in the matrix represents a paired comparison of two alternatives

2. cross out those ones which are double

3. start asking people: "What do you prefer, 1 or 2?" or "Which one is more important, A or B?"
and note down the answer in the respective cell

4. once you have received the respective answer, ask for the preference criteria: "Why do you
prefer this one" or "Why is this one more important?" and make sure to note down all answers
carefully

5. then continue to go through asking for all possible combinations and preferences

6. when all combinations have been asked and all cells are filled in, count the votes, each item
has received

7. write a list, ranking the items according to the number of preferences they have achieved

8. discuss within the group, whether the result of the Pairwise Ranking reflects the opinion of
the group

27
3.5.6 Wealth Ranking (Source: adapted from PL&A)

Description

Wealth Ranking (also called Wealth Grading) is a tool to support an analysis on the social
standing and conditions of certain groups of local people

Objectives

· to identify different socio-economic groups

· to investigate the impact of a specific intervention on those different groups

· to investigate the criteria of defining wealth amongst local people

Key Questions

- which different socio-economic groups do exist in the region

- what are the underlying rationalities for local people to define someone as ‘rich’ or ‘poor’
Facilitation

1. beginning by writing out the names of members of a community on cards. One name per
card, or one household per card.

2. The informant is asked to look at names on the cards and to identify which one is the best
off, which one the least well off.

3. This process is continued until all the cards have been identified.

4. The process can be continued with each of the three groups of cards that you now have -
best off, middle, and worst off.

5. The informant now has to be asked to explain what criteria was used to make these
distinction.

Once the criteria has been made clear, the team can go back and cross-check the information,
asking whether the criteria fits for all the ranked cards.

3.5.7 Semi-structured Interview (Source: adapted from Berg et al.)

Description

A Semi-Structured Interview or Informal Interview is an informative and relaxed dialogue, which

helps to find out information. It is a participatory tool that can be applied in a very flexible
28
manner. It is partly guided by the interviewer in order to find out information on pre-
elaborated topics. It is partly a casual discussion in which new aspects of a certain topic arise or
even new questions come up. It can be conducted with individuals or groups.

Objectives

· to gather information in a participatory way, giving the respondent the opportunity to

determine the topics to be discussed and the information to be given

· to allow for new questions to emerge

Facilitation

1. pay attention to the following principles:

- the interviewer and interviewee are partners

- try to make your interviewee feel as comfortable as possible

- be flexible; every new topic or idea can be useful, even if you were not asking about it

- listen and learn

- observe facial expression, body language, tone of voice, keep eye contact (if appropriate in
your working context)

2. pre-elaborate a checklist of the things you want to discuss with the interviewee

3. think carefully about who is appropriate to provide the information you are looking for, e.g.
individuals (to provide representative information), key informants (to provide specialised
information), groups (for general information), focus groups/interest groups (to discuss a
certain topic in detail)

4. choose the best and most convenient time and place for the interview

5. make your questions short and easy to understand - if possible use the local language - do
not ask more than one question at a time

6. try to avoid the following types of question:

- closed questions: questions which can only be answered with ‘yes’ or ‘no’, as they make
further probing for details difficult

29
- leading questions: questions that lead the interviewee to a certain answer (e.g. ‘Is it true
that...?’ or ‘Don’t you think that...?’

- ambiguous questions: unclear questions which can be understood in more than one way

7. for concrete information, use the Six Helpers: What? When? Where? Who? Why? How? -
however be careful, when using ‘why’ questions, because they may force the informant into a
defensive position and stop the flow of discussion - rather try to probe answers carefully with
e.g. ‘Suppose...’ or ‘Please tell me more about the reasons...’

8. record responses and observations fully

9. judge (for yourself), whether the response you get, is a fact, an opinion, or a rumour:

- fact: a common agreed truth in a certain place and time

- opinion: a person’s or a group’s view on a topic

- rumour: uncertain information from an unknown source

3.5.8 Seasonal Calendar (Source: adapted from Berg et al.)

Description

A seasonal calendar is a tool to explore seasonal changes (e.g. of climate, workload, grazing
areas being frequented, fodder and water availability) throughout the year

Objectives

· to generate information about seasonal trends

· to identify the time and amount of labour arising from activities done by women, men, elders,
youth within a group of local people

· to plan for additional/alternative activities

· to identify convenient time periods for meetings, training etc.

· to increase awareness of participants and field workers on different workloads at certain times
of the year

· to identify periods of particular stress and vulnerability, where support from outside would be
needed

Key Questions
30
- which are important milestones throughout the year of the local people

- are there free labour capacities during the year

- which individuals/groups would be in a position to contribute more labour during which time
of the year

- which might be the most appropriate seasons for additional activities

- which time constraints exist and what are the reasons

Facilitation

1. ask participants to draw a matrix, indicating each month down along the first column

2. encourage people to talk and discuss the items (e.g. main activities, movements etc.) to be
looked at, and to indicate them along the top row

3. then ask them to come up with changes in quantity/intensity of seasonally varying

dimensions and different actors in the respective cell of the matrix

4. then let them start together to identify potentials (e.g. in terms of resources/time) and
constraints throughout the year

5. try to find out, whether all these changes are similar every year, or if there are also
significant variations between the years and, if so, what causes them

31
IV ETHICAL CONSIDERATIONS

Generally, a researcher needs to think about the things they don’t want others to do if they are
asked questions. Here are some specific points to consider:

I. Participation and protection:

i. Ensure that you do not harm the people who take part in the interview or survey.
You don’t want to do anything that would cause physical or emotional harm to the
people you want to interview. This could be something as simple as being careful
how you word sensitive or difficult questions during your interviews.
ii. Consider the power imbalances and assess any possible dangers. When you are
doing research, be sure you are not taking advantage of easy-to-access groups of
people (such as children at a daycare) simply because they are easy to access. You
should choose the people you will interview based on what would most benefit your
research.
iii. Give respect to the interviewee/s or people you will ask questions You
should have the permission of the people who you will be studying to conduct
research involving them.

However, not all types of research require permission—for example, if you are interested in
analyzing something that is available publicly (such as in the case of commercials, public
message boards, etc) you do not necessarily need the permission of the authors.

II. Raising sensitive issues/confidentiality and trust :

i. Ensure there is follow-up to deal with sensitive issues which may be raised as a
result of the research.
ii. Be honest with the respondents, so that they know if confidentiality will ever be
broken and what the limitations are. Make sure that no confidential information is
made available to those who might misuse it. If necessary, make the personal
information unrecognizable (change names, places and so on). Many types of
research, such as surveys or observations, should be conducted under the
assumption that you will keep your findings anonymous. Many interviews, however,
are not done under the condition of anonymity. You should let your respondents
know whether your research results will be anonymous or not.

III. Conflicting agendas:

i. Some respondents or organizations or people from the community come to

participate in the research (maybe) with their own agendas. It is important to be
honest about these and negotiate differences.

IV. Informed consent:

32
 i. Ensure that the respondents know what they will be interviewed and that they have
right to say no to anything. The necessary consent is obtained to interview young
people or vulnerable groups.

V. About research

i. Respondents should know the purpose of what they are getting involved. You
should clearly explain to them how you will use the information they share.
ii. Be clear of how much time you need for this research since the beginning.

 Objectivity vs. subjectivity in your research is another important consideration. Be sure

your own personal biases and opinions do not get in the way of your research and that
you give both sides fair consideration.
 When reporting your results be sure that you accurately represent what you observed
or what you were told. Do not take interview responses out of context and do not
discuss small parts of observations without putting them into the appropriate context.

33
References:

I would like to acknowledge that this toolkit is developed based on the following manuals:

Toolkits: A practical guide to planning, monitoring, evaluation and impact assessment. By Louisa
Gosling with Mike Edwards. Published by Save the Children, 2003 UK

Did it Work? The Basics of Evaluation and Assessment for NGOs. By Valery Perry. Sponsored by
International Rescue Committee, NGO Development Program, Sarajevo, 2002 August, Bosnia
and Heregovina

34