Research Best Practice: Avoiding Confirmation Bias

One of the key skills in the sphere of market research and consumer insights is avoiding bias in the research. As we know, bias is a result of subjectivity. It’s the personal subjective lens used to view the world and is often based on certain intrinsic parts of character (gender, race, age, etc.) as well as socially-determined aspects like economic status, education, etc.

Confirmation bias is our natural tendency to interpret data in a way that is consistent with what we believe, our subjective lens. This subjectivity in research can be impacted by things like client interaction, previous work with similar brands, previous similar studies, or previous work with segments.

And this bias is harmful for analysts, moderators, and all stakeholders in the research process because it does not tell the full story, and might even get the story wrong. To avoid this type of bias (and start to rewire some of our own subjectivities), here are five ways to approach analysis and moderation:

1.     Identification of ambiguity.

While the meaning of some words may at first seem obvious, it’s important to remember that much of language comes with connotative meaning and subjective baggage. For example, “love” might conjure up very different meanings or associations for one person than another. In market research, it’s important that moderators and analysts not take any meanings for granted, lest our confirmation bias get in the way. Probing for further examples, synonyms, and details might actually prove that one respondent’s use of the word “good” actually just means “adequate,” while another’s might mean “excellent.”

2.     Don’t stop at what – ask WHY.

As with imbibing a word with positive or negative connotations depending on our own (or our client’s) ideas, sometimes quant data will tell us what we want to hear, so we might be tempted to stop at the numbers. At GutCheck, we encourage researchers to combine methods or work iteratively to better understand the WHY behind the numbers and, in turn, gain a more complete understanding of what a target audience is really trying to communicate.

3.     Read from all angles.

Along similar lines, it’s important to maintain self-reflection as a moderator or analyst and to hear (and acknowledge) all the voices in the conversation. Constantly challenge yourself to clarify and ask more from the research subjects if a viewpoint is unclear or in the minority of a group. As Dooley (Forbes) suggests, look for “alternative ways of interpreting words,” and hear “dissenting voices.”

4.     Hire an outsider.

There’s a reason so many companies and agencies hire vendors to conduct research, and it’s not just to save time! Often, those close to a project are naturally going to be most prone to confirmation bias – especially if the work on the line involves personal stakes. In these cases, though it may be painful to hear that an idea is not well received, it’s important to hire an outsider with no connection to the stimuli in order to more objectively read through data.

5.     Reviews and spot checking.

Peer-reviews and spot checking can be one of the best ways to read or listen to what’s truly in the research responses, and not just what we want to see or hear.  At GutCheck, our analysts work closely with the Online Research Strategist (ORS) team to understand client objectives, though they work independently to allow for a cleaner, more objective read through the qualitative data. On the other hand, the ORS is also monitoring a project to gain an overall understanding of the findings. In a system of checks and balances, the ORS proofs the analyst’s reports, and likewise, an analyst can provide feedback to an ORS to guide research writing or to question wording.  This two-person system allows for more objective research, and reduced chance of confirmation bias on either end.