Technical Report: Identifying and Removing Sources of Bias in Product Tests and Surveys

ABSTRACT

In product tests and surveys, bias occurs
when a measure from a sample systematically differs from
the population measure of interest. To put this into statistical
terms, a statistic is biased if it systematically deviates from
a population parameter, irrespective of the sample size. The
existence of bias determines whether one needs a control
product or item in a product test or survey.

There are numerous sources of bias that have been
identified. These include sampling bias, where a sample of
participants or the items to be tested do not represent either
the target population or the real test items. Bias also may
occur when participation or non-response in a survey is not
random, so that the opinions expressed do not represent the
target population. Leading questions, interviewer effects,
and uncontrolled individual differences can all contribute to
bias. Position bias and code bias are two sources that will
be discussed in this report. These two sources are relatively
easy to control but code bias, in particular, is often ignored in
practice. In the case of code bias, the codes themselves may
contribute to the responses selected. An extreme example
that we have observed involved data from a Chinese research
supplier. The Chinese ideogram for the number 4 is close
in appearance to that for “death” and therefore the number
“4” is often avoided in practical situations such as a floor
number in buildings and hotels. When one of the products
in their study was coded as “444”, it was not surprising that
it received a poor hedonic rating. There are many other less
dramatic sources of code bias that, if not controlled, may
lead to inaccurate parameter estimates.

This technical report appears as:

Ennis, D. M. and Rousseau, B. (2015). Identifying and Removing Sources of Bias in Product Tests and Surveys. IFPress, 18(1) 3-4.

Download this technical report here:
Identifying and Removing Sources of Bias in Product Tests and Surveys

Colleagues can download the list of 3-digit codes referenced within this technical report is available here:

3-digit codes

Not a Colleague? Click here to join for free!