Are Surveys Rigged Against Truth?

Ever been excited to take a survey on a topic close to your heart, only for the answer options on the first question to have you saying, "none of these speak for me"? This isn't just frustrating. It's disenfranchising. Why bother continuing if your true opinion isn't even an option?

This common experience begs the question: are surveys designed to gather real insights, or to reinforce preconceived narratives? If we can't see our perspectives reflected, the results are nothing more than manufactured data, skewed before they're even analysed.

Hans-Georg Gadamer, in Truth and Method, puts it this way: "We cannot understand without wanting to understand. Understanding does not occur when we try to intercept what someone wants to say to us by claiming we already know it." Yet how often are survey makers guilty of exactly this, deciding in advance what they expect to hear?

A case in point

Take the recent ESOMAR "Voice of the CEO" survey. Its question about methodologies categorised social media as a quantitative approach, denying any qualitative capacity. Arguably, quantitative analysis of social media is unscientific and misleading, but the framing reflects the common belief that dashboards are sexy. Such oversights not only skew the picture, they also discredit pioneers like Bakamo who are working to broaden the industry's understanding of social intelligence.

The larger point is a paradigm question. The way we design our surveys reflects our understanding of the space. The questions we ask frame the responses so they make sense and reinforce our frame of mind. When the ESOMAR survey implicitly denied the existence of what we do, it was not malice. It was a map that hadn't been updated.

To really address the methodological issue, we need to ask ourselves which research is better:

• A: where we select who is allowed to participate, or
• B: where everybody who has already answered is included?

I'll stop before launching into the power of natural conversations.

What happened next

I reached out to ESOMAR, and I had a lovely conversation with Joaquim Bretcha and Xabier Palacio. Even before the call they had already changed the survey, so it now includes an open-ended "other" option. So we went instantly from non-existent to an unspecified other. Not bad.

Even better, recognising that awareness is key, ESOMAR offered their help to raise awareness of qualitative social intelligence, so that eventually this approach gets its own tick on the survey. Proof that a public complaint, made in good faith, can move an institution.