You think survey research is dead? We've heard this one before...

In the realm of consumer insights, the humble quantitative survey – this workhorse of social science research – has been pronounced dead countless times over the years. Purportedly to be replaced with Big Data, CDPs, synthetic data, social listening, AI.

Well, insights fads come and go, but surveys have retained their utility. Yes, there are disadvantages to survey research (e.g. respondent bias, fraud, sampling issues). But these concerns are not unique to surveys. In fact, the chief reason that questionnaire-style research gets a bad rap is that it is such a difficult discipline to execute well.

Sure, at It depends we love digital trace data,¹ purchase behaviour, and so on. But 'found' data² are shallow, and they tend to reveal little about human motivations: beliefs, needs, and pains.

The key to understanding humans is to actually engage with them, and surveys remain a powerful tool for doing just that.

How to design a non-terrible survey

The output of a survey is only ever going to be as good as your research design. Once the survey is live, there's no going back. This is simultaneously a blessing and a curse. Although it sounds easy to 'just ask our customers a few questions', some people actually dedicate their entire careers to figuring out how to ask great questions.³

So, most businesses that end up disappointed with survey findings, do so because of poor methodology and poor planning, even if the idea of fielding a survey was sound.

Don't leave the house without your glasses

Your research has a goal, and that necessitates viewing the world through a suitable lens. If you go grocery shopping without your prescription eyeglasses, you may return home with a cucumber rather than a courgette.

In consumer research, your theory about the subject matter is your lens. Before diving into research design, you must explicate which expectations and specific hypotheses that you want to test. Simply casting a wide net and hoping for the right answers is not a great strategy. In most cases, you only have one shot at getting the questionnaire design right.

Designing your questionnaire means focusing this theoretical lens of yours. Your goal may be to measure a nebulous human construct such as 'medical self-efficacy', 'brand loyalty', or maybe 'trust in a service'. Asking respondents directly about their level of medical self-efficacy on a scale from 1-5 (or some such) is unlikely to yield any useful information, even from the 2% of them who happen to understand the question.

Instead, you need to apply a kind of weaponised beating around the bush: break the object of analysis into its constituent parts, and ask a bunch of individual questions that – taken together – triangulate the phenomenon.

Hey, phrasing!

Imagine this: Your survey results are finally in. You load the resulting dataset and begin compiling findings. Oh, horror! Some of these questions have no clear interpretation. Take this, for example:

I bet respondents were puzzled by this one...

Do these numbers mean that respondents who Agree and Strongly agree are satisfied both with the product and with the customer service? Why did roughly a third of respondents choose Neither agree nor disagree? Are they truly neutral, or did they just not know how to interpret the question? Some of these people may not even have encountered customer service!

Sweat leaks from your brow. What are you supposed to put in that slide deck for C-level? This is the dreaded double-barrelled question. Unclear or leading questions are the bane of an otherwise well-designed survey.

And how about this one:

This is what happens when you brainstorm response categories without any structure.

What on Earth are we supposed to glean from this? Some response categories overlap (price and discounts; ease of use and convenience). There's not much rhyme or reason to which aspects of the purchase decision were included versus left out. There's not even an option to select None of the above. Might we not have forgotten a few important categories as well?

All this, and we haven't even begun to unpack why it doesn't make any sense to ask about brand trust in this way. The reason why that category comes out at the bottom is probably that it lives way at the very back of the causal chain of decision-making, making it very difficult for respondents to even realise its impact, compared to much more practical considerations.

Keep questions simple, state them in plain language, and make sure that response categories reside at the same level of abstraction.⁴

Try not to get on your respondents' nerves

Poor user experience leads to survey fatigue, lower response quality, and increased dropout. Make sure that going through your survey does not feel overly cumbersome. Comply with accessibility standards, and – for Pete's sake – do add a page break every once in a while.

In particular, overly complex questions (e.g. large matrices, mandatory free-form text fields) take a toll on respondents. The overall length of the questionnaire can also dramatically impact completion rates.

Consider providing a small incentive for participation. Even something as simple as the opportunity to enter a prize draw can work wonders for respondent enthusiasm.

Take control of data collection

Getting input from a slice of your market is all well and good, but inferring beyond the sample is not trivial. You want to be able to claim that "this insight will generalise beyond the people we actually asked", right?

You need representativity: the sample of people completing your survey should resemble your population of interest as closely as possible. The gold standard approach here is random sampling. Picking respondents completely at random will produce a sample reflecting the underlying population on both observed and unobserved characteristics.⁵ Pretty cool.

In practice, however, unless dealing with public registries, CRM-based recruitment, or the like, you may have to make do with a less controlled approach: some kind of convenience sampling.⁶ If you are smart about it, you can minimise the types of bias introduced. This usually means employing sampling quotas, weighting data against an appropriate scheme, etc.

A nifty in-between solution may be to partner with a third-party survey panel provider. This can help provide a more structured access to your target audience, with quotas and weighting schemes in place.

On a related note: large sample sizes are often touted as the cure to bias. This couldn't be further from the truth. A larger sample size will only reduce the statistical uncertainty of your metrics – meaning that you may end up measuring the wrong thing really precisely.⁷

Know when to use a bit of magic

Say you are researching a sensitive topic. How to investigate attitudes and experiences that people would rather not reveal directly?

Or, you may need to establish a credible link between cause and effect, rather than obtaining purely descriptive stats.

In these cases, you need more sophisticated methods than just asking people outright. Examples of advanced survey techniques we have used:

• Identifying the most impactful version of messaging or framing using factorial designs.

• Settling on the best visual assets for ad campaigns using A/B testing.

• Informing product innovation or pricing strategy by using conjoint analysis⁸ to reveal customer preferences.

• Estimating people's propensity for dishonesty using digital games or simulations.

• Uncovering first-hand experiences with unlawful or undesirable behaviour using list experiments.

No matter how advanced your methods or your data sources, collecting reliable input from real people is tough to beat. Surveys are not the inferior choice – they just have to be done right.

Notes

1. See our recent post about using digital traces and online peer communities to track consumer behaviour in the pharma space: https://lnkd.in/p/em9hS_uF

2. Data sources not originally intended for research purposes, e.g. traces of online behaviour or discourse. Contrast with data compiled specifically for research, e.g. from lab experiments, questionnaires, etc.

3. Guilty as charged.

4. A great rule of thumb here is to keep categories MECE, i.e. Mutually Exclusive and Collectively Exhaustive.

5. Assuming no systematic respondent drop-out.

6. At its worst, this basically means getting a hold of whoever is easy to reach and willing to participate. This introduces a whole slew of biases in your dataset.

7. Reducing uncertainty is sometimes a goal in and of itself. Also, a larger sample size will better facilitate analyses of subgroups in the data.

8. Basically A/B testing on steroids: a randomised survey experiment which tests many different combinations of features at once.