Online surveys are just one of the many tools that public opinion researchers have to gather data. Thanks to a proliferation of DIY online survey platforms (with funny primate names), the mechanism is widely understood and used. As with most tools, online survey research offers a host of benefits and drawbacks that are important to understand if you’re going to get the most valuable, credible output. Let’s look at some of the most important pros and cons of online survey research.

The pros of online survey research

When looking at the pros and cons of online survey research, there’s a lot to put in the “pro” column. Online surveys tend to be less expensive, less time-intensive, and more accessible than other methods. These benefits allow organizations with smaller budgets to access the power of surveys when other methods are out of reach, and they allow larger organizations to gather more data more frequently. An online survey is a powerful, flexible tool that can be a great addition to your public opinion research. Here are a few of the main reasons why:

Less expensive to launch and sustain

• More “traditional” survey methods — including mail, phone, and in-person — require upfront and ongoing costs. Phone and in-person surveys are labor-intensive, requiring a large team of interviewers. Interviewers need to be trained and monitored when in the field. Mail surveys require printing, postage, and processing costs. In-person interviews, which are admittedly less common, require travel costs.  

• Online survey platforms are significantly less expensive. You have to pay for the distribution method of your survey (such as an email service, an online survey tool, or hosting on a website), but this is typically a fairly minimal cost compared to other survey modes. There are no rooms to rent, no reimbursements for travel, and no manual data entry.

• Past the initial setup, most organizations will spend far less to reach their target sample size. If you’re doing a phone interview, there’s usually a cost associated with each new response. With online surveys, there are increased costs with more respondents (leaving the survey up for a longer length of time, managing the additional data), but they tend to be much less expensive.

Faster

• Online surveys allow an organization to reach a large number of people in a short period of time. This is true even if you are targeting groups with specific characteristics (like people who have donated to a conservation charity in the last six months) or if participants are spread out across large areas.

• Less administrative time and effort. Researchers can send or post a survey once, then do something else while waiting for the results to come in. No one has to spend time reading the questions out loud to respondents or recording their responses. 

• Once surveys are complete, responses are transmitted or saved immediately. You don’t have to wait for someone to catalogue or record the results at the end of the survey process. You can skip straight to the analysis phase without any delays.

Design flexibility

• Online surveys provide a lot of flexibility and creativity in how questions get asked and information gets presented. Asking someone to watch a short video clip, for example, is much easier in an online survey than almost any other format.

• In some cases, design flexibility can reduce bias. Researchers have to manage many different kinds of bias when creating a survey. One of those is question choice bias (or “ordinal position bias”). This means that for multiple-choice questions, the order of the answers presented can affect the results. This is why good survey researchers switch the order of the answers whenever they do phone surveys. You can borrow this trick by programming your online survey to show possible answers in a different order to different survey takers.

Lack of an interviewer

• The fact that no human interviewer is necessary falls into both the pros and the cons of online survey research. Well-trained interviewers can make the difference in the sophistication and depth of thought that respondents’ answers present. However, in the case of a survey that asks for sensitive information (e.g., marital infidelity), the lack of an interviewer might reduce social desirability bias by allowing people to be slightly more honest, because of perceived anonymity and a lack of possible judgement.

The cons of online survey research

Even with all of the impressive time, cost, flexibility, and control benefits, online surveys have some potentially serious drawbacks. Many of them are contextual. If you can spot them, you’ll be more likely to work around them. Here are a few to consider:

Skewed demographics

• In general, people taking an internet survey will be younger, wealthier, and more educated than a general sample of the population. According to a 2019 Pew Research Center study, 10% of Americans don’t use the internet. Because that percentage doesn’t cut evenly across demographics, it can skew your results, sometimes significantly. For example, almost 3 in 10 adults with less than a high school education (29%) did not use the internet in 2019. 

Resistance to longer surveys

• More people abandon online surveys than phone surveys. This is especially true of longer surveys. Because online surveys have been around for a while, most participants are used to dedicating between five and eight minutes to the task. Sometimes, that simply isn’t enough time to get the rich, detailed data necessary for meaningful insights. When that happens, you’re more likely to see attrition. 

Trend data may be hard to match

• The Pew Research Center has shown that people respond to online and phone polls differently. If you’re starting your data collection from scratch, this one isn’t as important. But for certain segments of public opinion researchers, this one is especially problematic. For example, people who are taking an online survey are far more likely than those interviewed on the phone to give various political figures a “very unfavorable” rating when compared to people who were interviewed on the phone. This means that if you’re comparing phone interviews to online interviews, your results could be skewed. You may not be able to identify a genuine difference in public opinion and what could be ascribed to different polling methods.

Difficult to verify identity

• Unless an identification verification tool is used, identity is difficult to verify. Someone taking the survey could be a family member, a friend, or even a competitor who is purposely trying to skew results. If you don’t take steps to prevent it, online surveys make it more likely that one person submits multiple responses.

• Even with stringent identity verification, most online surveys will have a small number of bogus respondents. If you use a public opinion research firm (like us) to conduct your online survey, they’ll typically have robust protections built in to verify the identity of people taking the survey. This prevents repeat responses or responses from those outside your target group. Even with these protections in place, however, a 2020 study by Pew Research Center found that these online surveys may include as much as 4% to 7% bogus respondents. In addition, these bogus respondents tend to select positive answer choices – introducing a small, systematic bias.

Geographic Restrictions

• Online research is great for nationwide surveys. They’re also possible in larger states, but they can’t really be done in smaller geographies. If you’re looking to do a survey of a city, Congressional District, State Legislative District, or even a small state, it will likely be prohibitively difficult to find enough respondents.

Lack of an interviewer

• An interviewer makes it more likely that individuals will complete the survey and put some thought into their answers. Because online surveys tend to generate less accountability, people are more likely to answer quickly to get finished. With an interviewer present, it’s possible to ask follow-up questions, request clarification, and pursue interesting lines of inquiry that might otherwise stay hidden. Online surveys are solitary affairs, so crucial data is often missing from the results.

Getting the Best Results from Online Surveys

Why do you want to do an online survey? What do you want to know? This might seem like an obvious first step, but the most common cause of poor results (and therefore useless outcomes) is a lack of planning. For instance, a nonprofit that wants to know their current donors’ highest priorities should ask different questions than a nonprofit that wants to know how to get more young adults to donate.

Plan around your purpose. Starting with purpose can also help you decide if an online survey will be an effective way to get what you need. If you need quantitative data (data that’s countable), online surveys can be very effective. If instead you need qualitative data (data that gives you stories and rich, human insights), then an online (or in-person) focus group or some other research mode might better suit your needs.

Ask the right questions in the right way

Given the higher abandonment rates of online surveys (especially those over eight minutes or so), you need to be extremely pointed about the questions you ask, and how you ask them. Make sure every question ties directly into the purpose of your survey. 

The phrasing is equally important. Even subtle changes to the questions in a survey can result in varied, even biased data. To have confidence that you’re creating a survey that will eliminate the chance of bias, consider:

• How the questions are worded
• Where the questions are asked in the survey
• What answer options are offered (or not offered) to respondents
• What context is provided (or not provided) to the respondents

Use a blended approach

Many organizations feel like they have to choose a single research tool, but that’s not always the case. Often, using multiple research tools can give you the most useful data. A blended approach can balance the strengths and weaknesses of various modes, giving you data you can use with confidence. For example, we often combine online surveys with qualitative research modes (such as focus groups) to get a more well-rounded feel for the issue. 

A blended approach also allows you to access both quantitative, numerical data and qualitative, story-rich data. This combines the power of hard data you can collate, with the benefits of nuance, story, and context.

Talk to an expert

The DIY nature of online surveys, the simplicity of the platforms, and the speed with which you can gather large amounts of data all make the tool popular.  But they’re far from perfect. Rather than taking the time to identify and mitigate the potential drawbacks of an online survey, many organizations will throw a survey out into the online world and hope for the best.

We’ve seen hundreds of terrible surveys happen that could have been prevented with proper planning. More importantly, we’ve seen the disappointment of having put forth effort and budget to yield useless results. As public opinion researchers who have been in the business for decades, we have the institutional expertise to do every step right. We’ll work with you to create a survey approach that gives you the accurate data you need to get the best possible outcomes.

What do you need to know? We can help. Get in touch or subscribe to our newsletter.