How bad survey design can discourage people from answering your questions

November 25, 2007

One of the things I’m learning a bit about is survey design and statistics. There isn’t going to be anything new, here, though, so you’ll appreciate this even without a specific background in these fields. And namely: if you ask people wrong questions, you’ll get wrong/bad/no answers.

Let’s look at this survey from Opera that is shown after you download their browser.

What’s bothering me here is the second question. “Why do you use Opera instead of another browser?”

What makes you guys think that I do it instead of another browser? I regularly run 4 browsers (Safari, Firefox, Opera, IE, in approximately this order of frequency). In fact, the three first ones are running as I write this, as I like different things about each. Even if you argue that most people only have one browser on their PC, one of the options in this very survey is that you may run Opera on your mobile and something else on the PC.

Surveys are always about generalizations, sure. But this is a bit too much and throws out a lot of nuance exactly with the community that Opera should embrace – the techies who regularly run different browsers and can appreciate its advantages.

As a solution to this question, it would have been a great place to ask, “What other browsers do you run and why?” Have people rank features of browsers against each other if they really run several. Saying instead of was an immediate reason for me to not even bother with this survey further down, who knows what other bad questions I might have to waste my time on.