Research Process & Survey Methodology
Before you define your methodology, it is important to understand the two types: Qualitative and Quantitative. Each type has its own benefits and disadvantages. Usually, it is important to combine both qualitative and quantitative methods to conduct professional research.
In our case, Community-Based Monitoring (CBM) is a type of large, constantly evolving research project with practical applications that uses both qualitative and quantitative methodologies. For the past 5 years, we have been conducting A LIVE EXPERIMENT in a unique social system, where the actors, provinces, projects and scale of monitoring keeps on changing. However, the underlying methodology of our program remains the same.
After extensive study and desk research, the founders of the CBM approach presented 3 hypotheses:
If community members are directly engaged in monitoring infrastructure projects in their districts, then the quality of international donor-funded projects will improve.
If community members are directly engaged in monitoring infrastructure projects in their districts, then the accountability relationship between the citizens and the government will improve.
If community members are directly engaged in monitoring infrastructure projects in their districts, then local monitoring data will help donors, government ministries and others improve the implementation, monitoring and bidding processes for infrastructure projects in the future.
are the elements, individuals, groups, institutions, social bodies, and countries that interact and are part of a social system.
is the way that actors interact with one another in a system. The system is created by a set of rules, certain historic patterns, historic evolutions, behavioral patterns and resources available.
is a process of testing a hypothesis that you have begun without knowing exactly what the outcome of these actions will be. Community-Based Monitoring is still in experimental phases because we do not know how the program will evolve and what are its long-term consequences.
In the Community-Based Monitoring (CBM) program, we collect various types of DATA. After the local monitors are trained, and before they start monitoring, they conduct a:
BASELINE SURVEY: Quantitative
WHAT IS A SURVEY?
A survey is a quantitative research tool that describes a population. It counts and describes its particularities. The surveys were initially used to predict population’s needs and understand demographic changes.
Surveys can describe phenomenon, but not explain them. They are thus, a statistical not an analytical tool.
The whole population is not usually surveyed. Rather, a representative sample of the population is selected. Using statistical models, it is possible to choose a percentage of the population from which you can generalize patterns and perceptions of the larger population.
In the CBM program, we decided to add a Baseline and Endline Survey in our program methodology to produce data that helps us understand the following issues:
1. Understand the Level of Transparency & Community’s Access to Infrastructure Project Information.
2. Understand the Level of Community Satisfaction with Project Delivery.
3. Understand the Community Perceptions of Accountability Issues in Infrastructure Projects.
GETTING A GOOD, REPRESENTATIVE SAMPLE
Sometimes in a population, there are certain groups that are under-represented in surveys. In Afghanistan, women are often an under represented population. If women make up 50% of the population, then during a survey maybe only 40% of the respondents will be women.
If the sample does not accurately represent the population, then is it possible to still generalize results from the flawed sample?
Yes, by WEIGHTING the sub-groups within the sample.
SURVEYING IN AFGHANISTAN: A FEW NOTES
SECURITY A surveyor has to be extremely careful when conducting an interview and in certain cases, especially if the survey is about corruption and/or integrity issues, questions have to be ‘handled’ with care. Due to the nature of the questions posed, IWA surveyors are sometimes physically threatened in provinces like Farah and Kandahar. The surveyors thus, have to leave the area and continue the survey in other districts. In some cases, surveyors have been arrested by local authorities who do not know that surveys are permitted. It is thus, advisable to first get government permission in Kabul and in the provincial capital.
LIMITED ACCESS TO WOMEN Women should be surveyed in equal proportions in order to maintain the statistical significance of the sample. In some provinces, it is helpful to use female surveyors to interview women. A letter of introduction from the Directorate of Women’s Affairs in the Province can help when all surveyors are male. It is also possible to gather women in one location, in order to reduce the travel and security risks. Women can also be accessed through phone surveys with female interviewers. However, even with some of these precautions, some families may still not allow women to respond to surveys.
PHYSICAL ACCESS Lack of roads and snow can limit access to certain survey populations. In these cases, time management is essential in order to avoid delays caused by the limited access to remote areas. It is also important to have enough questionnaires printed in advance. Techniques like PDA can be a good solution if there is a possibility of recharging them.
HIGH LEVELS OF ILLITERACY pose some problems for surveying in Afghanistan. Questionnaires are usually completed by the respondents themselves, but in Afghanistan due to the high levels of illiteracy it is normally the surveyors who fill the survey forms. In this case, it is extremely important to formulate clear and simple questions. In some cases, like in remote areas, the answers to the question ‘How did you access information on the project?’ was ‘Through the Internet,’ which is a clear case of an impossible answer. In such conditions, the surveyor has to be well-trained to not accept impossible or illogical responses. The surveyor should also avoid influencing the respondent’s answer in any manner.
BASELINE DATA OF THE POPULATION ARE NOT WELL ESTABLISHED, which means that we do not have accurate data on subgroups, minorities and regional demographics in Afghanistan. To avoid this problem, Integrity Watch gathers the most recent information on major surveys and a study conducted in Afghanistan and examines their sample. Integrity Watch uses this sample as a base for selecting its own. The idea is to be consistent in order to have a study coherent with one type of population sample so that the results of our surveys evolve in the same manner as the population.
ANONYMOUS Due to the sensitive nature of many surveys, it is important to preserve the anonymity of the respondent. Anonymity also helps ensure that data analysis is not biased by knowing the name of the respondent. Collecting enough demographic information helps test whether a survey was conducted properly. When people are willing to give personal information, the data is helpful to understand why portions of the population behave in certain ways. However, a surveyor should never force or try to convince people to give personal information.
REPRESENTATIVE SAMPLING METHODOLOGY
DETERMINING THE RIGHT SAMPLE SIZE
THE LARGER THE SAMPLE SIZE, THE SMALLER THE MARGIN OF ERROR. We should be very careful in selecting our sampling to avoid large and unacceptable margins of error. If our target groups consist of different categories, then we should also broaden our sample size to represent all the categories. For example, if we want to conduct a survey among different people who received various trainings (at least 15 types of trainings) then we should select our survey respondents from all those 15 categories USING A STRATIFIED SAMPLE.
For a national survey covering 50 million persons a day, the sample can have as little as 500-1000 individuals if it is a representative sample. However, in order to survey a small population such as a class room, of 38 individuals with a lot of diversity, the sample should have at least 20/25 persons to be representative. For a random sample the sampling error depends on its size, the bigger it is the most accurate the results will be.
But from the equation, we can see that doubling the size of the sample population will NOT halve the margin of error. Thus, even if the population is 100 million, we do not need many more than 500-1000 individuals for a representative sample.
*Note: Always write down the data source used in your report, so the reader is aware about the possible limitations faced in the survey. This is the case especially if weighting techniques were used to compensate for under-representation of certain parts of the population in the sample.