The questionnaire drafting process is one of the most critical parts of a project.

Before a (good) polling firm fields any survey, they will review and revise the survey questionnaire multiple times, until the final version is a carefully crafted instrument. The questionnaire drafting process is one of the most critical parts to the success of the project. It’s when pollsters decide not only what to test, but also who, where, when, and how to test. 

The “how” covers the type of survey used, the order of the questions, the specific wording of each question, and the order of the responses. These factors are especially important for accuracy — and they tend to be the hardest to get exactly right. Our team goes through each questionnaire with a fine-tooth comb, assessing every word on the page. This may sound like an exaggeration, but we really have spent hours debating a single word. And for good reason. Developing a questionnaire is a science, and psychology plays a profound role. Understanding how people think and answer questions is imperative to developing a questionnaire that avoids the weaknesses and biases of our brains to arrive at meaningful, accurate survey data. 

How People Think

In his book Thinking, Fast and Slow, renowned psychologist Daniel Kahneman explains that when people process the world around them (or when people think), there are two distinct systems at work. The process he calls “system 1” is “fast, intuitive, and emotional.”¹ Think of this as your gut reaction. It’s what you use if you want to solve 2+2 or read this simple sentence. “System 2” is “slower, more deliberative, and more logical.”² It’s what you use when you fill out your taxes or drive your car during rush hour on a busy highway (or at least it is if you drive safely). Our system 1 operates automatically without intentional thought, while our system 2 requires effort and attention. These two systems work together, but there’s sometimes a conflict between which system dominates. Because our system 1 is automatic, it’s usually first up to bat when we think. System 2 is only called on when deemed necessary. To put it bluntly, our brains are inherently lazy, seeking the easiest answers to the most complex questions. And often this laziness leaves our brains susceptible to biases and prone to errors.

How Pollsters Think

Being aware of these weaknesses of our brains is important when we consider the ways we ask questions in our surveys; or if you will, we must activate our system 2 brain when considering how our system 1 brain might improperly apply itself in answering our survey questions. We’ll look at two examples: priming and the halo effect.

Priming
The first example is understanding the ways our thoughts can be unconsciously primed. That’s what happens if I present the word EAT to you and then I ask you to complete the word fragment SO_P. Psychologists have found that you are more likely to have thought the word was SOUP rather than SOAP because the idea of the word EAT temporarily primes the idea of SOUP. As Kahneman explains, “psychologists think of ideas as nodes in a vast network, called associative memory, in which each idea is linked to many others. There are different types of links: causes are linked to their effects (virus -> cold); things to their properties (lime -> green); things to the categories to which they belong (banana -> fruit).”³ 

As survey researchers, we are hyper-aware of the unconscious priming that can occur throughout the course of a survey and put many safeguards in place to counteract this issue. For example, if we’re trying to gauge the audience’s opinions of a candidate, we test that in the beginning of the survey, before any other question about an issue or ballot. Doing this ensures that reactions to the candidate are not merely an ancillary reaction to another issue we had mentioned moments before. Another example is when we want to know how many people know the date of an election. In this case, we don’t give any indication of the election’s timeframe at any point in the survey before that question is asked. 

The associative memory mechanism that causes unconscious priming can’t always be avoided; in fact it informs all our opinions about everything. But we can lean into the associative memory mechanism and turn it into a useful tool in our research. To this end, we often start out focus group discussions by asking participants to independently map out on a sheet of paper a word cloud of all the thoughts and reactions that first come to their minds on a given issue. This activity forces the participants to think deliberately about a topic, encouraging system 2 thought and helping uncover the unconscious priming effects at play. Acknowledging the important role of associative memory and priming is a critical step in understanding reasons behind audiences’ opinions.  

The Halo Effect
Another common bias our unconscious (system 1) brains fall victim to is placing an increased weight on first impressions, which is called the halo effect by psychologists. To illustrate the halo effect, here’s an example Kahneman uses in his book. Read the following descriptions of two people and think about what sort of personalities these people might have:

Alan: Intelligent—Industrious—Impulsive—Critical—Stubborn—Envious

Ben: Envious—Stubborn—Critical—Impulsive—Industrious—Intelligent

After those descriptions, what do you think of Alan and Ben? Psychologists have found that most people will view Alan much more favorably than Ben. In fact, they’ve found that “the initial traits in the list change the very meaning of the traits that appear later. The stubbornness of an intelligent person is seen as likely to be justified and may actually evoke respect, but intelligence in an envious and stubborn person makes him more dangerous.”⁴ 

The takeaway for survey research here is that sequence matters. When we test, for example, a series of messages, we want to ensure that the order that participants hear the messages don’t influence their reactions to it. So we rotate whether the respondent hears positive messages or negative messages first and we randomize every single message. Or, if we’re testing the ballot for a political race, we randomize the order that candidates’ names are given. These randomization and rotating measures create a level playing field between the topics we’re testing and reduce any undue weight given to what’s read first.

Better Questionnaires = Better Results

There are many important and hidden ways a survey questionnaire can impact the results. If it’s not done properly or with an appreciation for how we think and answer questions, it can lead to inaccurate results or misleading findings. Understanding the myriad ways our psyches can be influenced is key to ensuring biases do not show up in our final data. As survey researchers, we are keenly aware of these factors because we want our surveys to give a voice to the populations we’re researching and to give everyone a clear picture of public opinion untouched by any researchers’ fingerprints.

---

1. Kahneman, Daniel. Thinking Fast and Slow. 1st ed. Farrar, Straus and Giroux. 2011. Page 20-28.
2. Kahneman, Daniel. Thinking Fast and Slow. 1st ed. Farrar, Straus and Giroux. 2011. Page 20-28.
3. Kahneman, Daniel. Thinking Fast and Slow. 1st ed. Farrar, Straus and Giroux. 2011. Page 52.
4. Kahneman, Daniel. Thinking Fast and Slow. 1st ed. Farrar, Straus and Giroux. 2011. Page 82.