“You got your chocolate in my peanut butter!”
So goes the Reese’s Peanut Butter Cup ad in which two great tastes are miraculously combined, to the delight of partisans of both chocolate and peanut butter.
It’s not quite that simple in science, where the team of labcoats who really, really like chocolate don’t exactly get along with the team that likes peanut butter.
In this case, the chocolate is a proxy for hard numbers--experiments, surveys, big data. We call this quantitative research. The peanut butter is squishier stuff--observations, interviews and participant observation. Both of these approaches are valuable to developers, but what keeps them separate are the scientists themselves.
Why? This is a left-brain/right-brain issue. Some people are more drawn to the logical and others to the creative. If you’re a game or mobile app developer, you recognize that some of your staff are ace programmers, some are ace artists, and the overlap is pretty rare. This is true in every industry.
The quantitative stuff is almost always given more weight than the squishy interview work. After all, who can argue with numbers and graphs? Yet there’s a risk in avoiding the squishy. Here’s why.
Qual and Quant work essentially answer two different sets of questions. If you only care about one set, great, but in truth, most of us care about both sets.
Set 1: Why did this thing happen? What does it mean? How does it fit in with the larger culture, community, etc?
Set 2: Is this common? How much of it is there?
Now you might jump right to the first set as the most important, and it may surprise you to hear that this is the province of qualitative work. This is where interviews and observation shine, and where surveys often fall flat.
The difference between the questions and the methods really comes down to sampling and depth. Sampling is the process of picking some cases so that your results are likely representative of the whole population. Quant work is really good at this. Depth, on the other hand, is about finding deeper truths and understanding.
The rub is that quant work often ignores depth, and that qual work often skips right past representativeness.
So, what’s the solution? Realize that these are different question sets, and that they are complementary. The best understanding of consumer behavior comes from the triangulation of both, elspecially in game analytics. As a metrics provider, I recognize that automated analytics systems are almost never going to give the client depth and meaning. So, it’s worth putting in some time and effort to directly talk to the end-users. A survey is a start, and interviews and observation are the next level.
Often the qualitative researcher will find phenomena that no one had considered. Then, it’s the quant’s job to find out whether the phenomenon is actually common, and whether it’s impacting the bottom line.
In the end, the two flavors really do taste great together.
For reference, there’s a much deeper version of this issue (for games research) spelled out in a paper I wrote, here.