No social or cultural research is completely free from bias. Surveys and focus groups are typically answered only by certain types of people, and therefore don't fully represent the range of beliefs in the marketplace. Interviews rely on a small sample size, making large extrapolations difficult, and often involve people who are being paid to participate. Rewards further complicate these traditional methods. Mixed-method approaches require judgment to link qualitative and quantitative results and are therefore subject to bias.

The point is, nothing is free from bias.

Lux for Insights Leaders holds that our approach to data has less bias than other methods. Here’s how we think about the major sources of bias big-data based approaches face.

How do we minimize bias?

1. Relying on extremely large N samples (big data from millions of consumers). This is a proven way to minimize bias as shown through the Nobel Prize-winning framework of Prof. Daniel Kahneman (creator of Behavioral Economics).
2. Building an industry-leading methodology of data gathering and machine-led contextual analysis that allows us to ensure that inherent biases at the platform level are minimized.
3. Constant reassessment of our data sources by our team of anthropologists to constantly find ways to get our sample set as close as imaginable to the national distribution.
4. Building a mathematical model (that runs the demographic calculations) that is best-in-class, designed by experts in statistical modeling in the computer and social sciences.

Aren’t you only studying internet users?

Yes.

However, internet users represents a far bigger swath of the population than you think.

Who Uses the Internet?

Here's a quick summary of the breakdown of social activity by age in the United States.

It shows that with age comes lower activity, but even among the 50-64-year-old consumers, it's still at a whopping 64%.

As a comparison, the percentage of Americans between the ages of 50-64 who regularly take surveys is less than 5%.

Next, take a look at the graphic below, which outlines where consumers spend their time when on the internet.

Again, this is divided by age: light blue is 18-29. Dark blue is 30-59. And Grey is anything over 59.

If we focus on the 60 or younger crowd, we can see that the distribution of internet activities is relatively even. The younger group tends to use instant messaging, social networking, and video content more, while the older crowd over-indexes on news sites and web searches (which often lead to blogs, news portals, and forums). Even for platforms that are popular among younger audiences, such as social media, the divide between the 18-29 and 30-59 age groups is not as significant as it used to be. (Note that this data is from 2017 and the divide has likely lessened further, partly due to increased smartphone use.)

Isn’t each website generally dominated by one socio-demographic group?

1. This is why we collect data from all kinds of platforms - forums, news sites, blogs, YouTube, product review sites, and e-commerce sites.
2. We know of the skews inherent in certain data sources. It is for this reason that we apply various corrections at the level of data input each month. This allows us to ensure that the input data closely resembles an even distribution of the US population from 18-64.
3. Our mathematical models are tested and validated through hundreds of experiments, both internal and external. Our clients have many times pitted our results against those gathered through quantitative means and have found a difference of less than 5%. This, given the expediency and level of detail inherent in ethnography, is a massive gain for any client who'd otherwise get mathematical results with no context or understanding of intrinsic human motivation.

Does online culture reflect reality?

We have a terrible habit as human beings to dismiss things that seem unfamiliar or uncomfortable to us. So, if we're not comfortable with the study of culture on the internet, we tend to brand it as something only relevant to the internet. That argument might have held up in the early 90s when a significantly smaller and highly educated and tech-savvy audience was online. But in today's world, that argument does not stand.

We are now hyper-connected through our cell phones. There isn't a single cohort of consumers that one cannot find or study on the internet. Why? Because through pseudonyms and text-based engagement, consumers feel they can open up on the internet in ways that they cannot in person. They can truly express their beliefs, values, fears, without fear of backlash from their loved ones. They may experience backlash from strangers, but there's a level of personal separation that creates honesty in online discourse and makes it by far THE RICHEST source of information on human beings and their beliefs about pretty much anything and everything.

When we conduct a survey, we don't say "oh that represents survey culture." We are more than happy extrapolating the data from 1024 respondents to the entire population, but we struggle with a study that observes tens of thousands of online consumers. Why? Because it's new or different, and that makes us uncomfortable. The more we recognize that, the easier it will become to accept the hard facts about the value of the internet to research in the modern world.

(PS: We're not even going into the merits of observational data vs. respondent data here because that is a whole other set of advantages that the internet serves up.)