The perils of straying from protocol: sampling bias and interviewer effects

PLoS One. 2015 Feb 18;10(2):e0118025. doi: 10.1371/journal.pone.0118025. eCollection 2015.

Abstract

Fidelity to research protocol is critical. In a contingent valuation study in an informal urban settlement in Nairobi, Kenya, participants responded differently to the three trained interviewers. Interviewer effects were present during the survey pilot, then magnified at the start of the main survey after a seemingly slight adaptation of the survey sampling protocol allowed interviewers to speak with the "closest neighbor" in the event that no one was home at a selected household. This slight degree of interviewer choice led to inferred sampling bias. Multinomial logistic regression and post-estimation tests revealed that the three interviewers' samples differed significantly from one another according to six demographic characteristics. The two female interviewers were 2.8 and 7.7 times less likely to talk with respondents of low socio-economic status than the male interviewer. Systematic error renders it impossible to determine which of the survey responses might be "correct." This experience demonstrates why researchers must take care to strictly follow sampling protocols, consistently train interviewers, and monitor responses by interview to ensure similarity between interviewers' groups and produce unbiased estimates of the parameters of interest.

Publication types

  • Research Support, Non-U.S. Gov't
  • Research Support, U.S. Gov't, P.H.S.

MeSH terms

  • Adult
  • Effect Modifier, Epidemiologic*
  • Female
  • Humans
  • Logistic Models
  • Male
  • Research / standards*
  • Research Personnel / standards*
  • Selection Bias*

Grants and funding

This work was primarily done on a voluntary basis by two graduate students (studying at Kenyatta University and the Johns Hopkins Bloomberg School of Public Health). The students were sponsored as graduate student interns in 2006-07 by the Centers for Disease Control and Prevention in Kenya, meaning that they were provided with workspace and guidance. CDC/Kenya paid for the staff and transport costs to implement the survey. The funder therefore had a role in the study design and data collection phases. Subsequently, data analysis and discussion of the findings happened without formal CDC/Kenya involvement. The students continued to correspond with their mentors who had formerly worked with Johns Hopkins and CDC/Kenya, though the students no longer had a formal relationship with either institution. The students consulted these mentors on the preparation of the manuscript and the decision to publish.