Toolkit

Issues of bias apply equally to context analysis, baseline studies, evaluation and monitoring and should be addressed as early as possible in the design process of any programme and M&E framework.

Avoiding researcher bias:

Conduct exploratory research and review any available similar surveys to help refine your strategy and test for bias before conducting the full survey. Bias can occur after collecting data at the analysis stage. To reduce this risk, anonymise data and create a data analysis plan before you write your survey and include peer review mechanisms.

Choosing a sample:

Identify your sampling strategy early in the survey process. Make sure you clearly define who your respondents are during design phase and, where possible, over-sample to compensate for attrition (lower response rates) and removing incomplete data. It may be difficult to have a representative sample, as some groups may be more willing to engage than others. For example, in some contexts, women might be more likely to participate or young, well-educated and engaged youth might be more likely to participate than other young people. Understanding the population and having a strategy for weighting of data should compensate for this.

Dealing with response bias:

Response bias can occur when response rates are low as data is collected from those who respond to surveys (those who have self-selected to respond). Response bias also refers to the inaccuracy in answers given by respondents. Issues can be acquiescence bias, when respondents say what they think you want to hear, and desirability bias, when respondents ascribe to behaviours and characteristics that are desirable. Online or anonymous surveys are ways to reduce such bias, as well as asking neutrally worded questions.

Triangulating data collection methods and data sources:

In PVE, programming data collection can be challenging and available data limited. In addition, reliance on a few data collection methods can increase the risks of bias within data collected, as the biases can be amplified in a PVE context. Using a number of different methods to assess change can reduce these risks, and various methods can help contextualise and validate data, as well as highlight gaps or limitations.

What to think about when choosing sampling methods

There are well-established practices on how to sample data to enable generalisations to be made about the data. Larger samples tend to give more accurate results but this depends on what sampling methods are chosen. For generalisations on a wider population to be valid, a sample has to be randomly selected.

Random sampling techniques include the following:

Simple random. The simplest sample for statistical inference is a random sample of the whole population.
Stratified random. This is a sample based on selecting people at random within pre-specified groups or categories, such as age group, ethnicity, or gender.
Cluster. In cluster sampling, groups of people are used as the ‘sampling units’ instead of sampling individuals. For example, schools might be sampled instead of individual pupils.
Systematic method. This relates to selecting people in a systematic way rather than at random. For example, you could sample all the odd houses on a street, people who have their birthdays on a Monday, or people with surnames starting with every other letter of the alphabet.
Multi-stage. This approach involves sampling in more than one stage. For example, if you begin to sample regions in a country, then sample districts within each region, then individuals from each district. Multi-stage sampling can be combined with other approaches such as cluster or stratified sampling.

There are few examples of random sampling being used in PVE programming, considering constraints within PVE contexts. Non-random sampling approaches may be more suitable:

Convenience sampling. This involves asking people who are available or able to participate without formal sampling.
Purposive sampling. This involves selecting a sample to represent specific characteristics of a population (e.g. young people living in urban areas).
Snowball sampling. This involves sampling where the individuals or organisations invited to participate in research are themselves invited to identify other people who might be eligible to participate. This is a very good approach for recruiting so-called ‘hard-to-reach’ populations, e.g. extremists or gang members, who may otherwise be difficult to recruit to an evaluation.
Quota. This involves setting quotas for each characteristic in the sample. For example, the sample might require that the same number of people be sampled from a specific age group.

Monitoring strategy And data collection

Module 6: Data collection methods

6.3 Tips for dealing with bias

Avoiding researcher bias:

Choosing a sample:

Dealing with response bias:

Triangulating data collection methods and data sources: