Scientific Method and Video Game "Studies"

07/21/2015 - 12:39 | By: Kindra Pring

Video games are certainly a fascinating subject aren't they? As a relatively new phenomenon, they have become the interest of many scientists and researchers. Why not? Video games not only offer a very unique way to study human behavior, but are effective tools in themselves. The same programming used to make video games can be used to make research tools to better understand all kinds of things, from psychology to physics. You would think, given their usefulness, that video games and science would appreciate a friendly relationship, characterized by fairness and mutual respect.

Which makes it all the more insulting when science seems to use its time constantly attempting to sabotage games even if it means sacrificing its integrity in the process. Researchers seem to take abject pleasure in pushing the idea that video games are the absolute worst thing to ever befall society. If you look up studies, you'll find they tend towards focusing on proving that video games cause everything from violence to sexism to mental disorders. What you'll also find is the peculiar pattern that these studies seem to assume that going in. That specific journals are prone to never publishing anything that doesn't completely support that pre-conceived conclusion. And that apparently objectivity takes a back seat to padding your research councils on video games with people who have clear biases towards the "video games cause terrorism" motif.

Indeed it seems if you look outside of the published journals you find a lot of established scientists pointing out that the conclusions peddled by these "studies" aren't as clear cut as organizations like the American Psychological Association would like you to believe. So why would scientists try so desperately to sabotage an industry that has provided so much to progress and discovery? There are a million conspiracies you could draw from that, but that's not why we're here. Most recently, many journalists have been happily pushing a new "study," which proclaims that players who aren't as skilled are more likely to be sexist.

A few things to note about this study. First, the skeleton is a direct rip off of this study, conducted by Nicholas Matthews. I say rip off, because despite the fact that very basic procedures are nearly identical—and the behavior being studied very similar—the author doesn't mention the study once. This could be coincidental, though it should be noted that the author of the original study took a very positive view of gamers in his piece, concluding that aggression is caused not by games themselves but by frustration caused by a lack of ability. It is readily available to anyone with access to basic research libraries and was peer reviewed and released in a fairly reputable journal.


However, even though the basic concepts are the same, that is about all these studies have in common. While the original study was objective, fair, and professional, this study is clearly affected by the biases of the researchers, which makes the highly questionable methods even worse. Rather than creating their own game, allowing the researchers to control the environment, Kasumovic and Kuznekoff used Halo 3. Rather than a carefully designed sample method, they used a sample of convenience based solely on their randomly encountered counterparts. Rather than a consistent measure based not only on observation but self-report, they use a highly unreliable means of determining skill and behavior, one which could be very easily tainted by pre-conceived biases.

You can't completely blame the researchers though. A large part of the problem with this study is how it has been portrayed in media. That is, as absolute and irrefutable fact. Let's have a talk about the scientific method, Internet.

First, when conducting an experiment of this kind, you must control for as many factors as possible. This does not simply mean having a control group. For my Senior year of college, I had to conduct a very simple study to practice methods learned in an undergraduate research class. By nature of my status as an undergraduate, the study wasn't going to be published; was going to have a small, convenience based sample; and likely no one but me, my associate professor, and maybe the professor himself would see it. Even then, I had to control for a lot, and anything I couldn't control for, I had to explain. Were any of the survey packets different? Was the room too hot? What was I doing? What is the possibility a person's answer is impacted by their mood at that time? Especially when studying people, there are countless things you have to account for. This study doesn't account for any of them. It doesn't even do the basics of tracking demographics of users.

This is why the original study went as far as having their own game designed. When studying video games it may seem like it's an easy thing to control, but there are more factors as play. Especially if the game is played online. You aren't witnessing the person in the other room, you can't see if anything around them is affecting them or their play. You don't know what their connection is. You can't even be 100% positive of the gender of the person on the other side.

Second, sample is important. Not just the size, though obviously that is quite important (for the record, despite having such a lazy design, this study has a pitifully small sample). How did you select your sample? The original article used a valid selection process with true randomization. This study used a sample of convenience, usually reserved for students and testing instruments. Essentially, this means the researcher set up shop, then took whoever walked by as their subject. There is no rhyme or reason, and while it may seem random, mathematically and psychologically it isn't. Halo 3's servers aren't random. They use an algorithm to match players against each other. And this algorithm can be influenced by past games. Essentially, this study is designed so that the actions of past subjects can easily influence the actions of future ones. And it isn't accounted for by any means.

As a third point, this study is very questionable in its following of informed consent rules. Science, particularly psychology, has not had the most sunny history in terms of participant treatment, so there are many stringent rules in place regarding consent. The researchers actually do address this, stating that consent was not collected because users were anonymous. However verbal consent is allowable and no rules strictly state whether anonymity plays a factor. Technically speaking, all participants should be treated as mostly anonymous. They also state that because users pre-agreed to games being recorded due to accepting the XBox Live Terms of Service, that this counts as exempt under Category 4 of the Institutional Review Board.


But consent means more than just consent to be recorded. Consent requires that the participants know that they are being studied. You do not necessarily have to say why in the beginning—a practice commonly used in experimental studies to avoid participants altering their behavior to fit what researchers expect—but you also have to do a debriefing at the end of the study and give participants full information. None of the justification given covers any of that, because users may agree to being recorded, but that does not mean they agree to being studied. Critically, this article is not only poorly made, but possibly unethical.

And lastly, this one specifically to the media harping this as the absolute truth: science is not a one shot deal. Particularly not psychology. It can take hundreds of studies before scientists can collectively agree that something is essentially fact. One study does not decide human behavior, especially a study as badly constructed as this.

Taking all this into account, it is frightening how seriously people took this study. The paper is littered with statements like these: "This does, however, reinforce the fact that women are entering a very male dominated environment" in inappropriate places like the Results section, typically used only for the dry data, not commentary. Completely ignored by the media is the fact that women are also treated better by high skilled players—apparently by a much larger margin than male players are treated. If you look at the raw numbers, you find this supposedly epidemic problem is represented only by 11 people, and the researchers had to specifically manipulate this just to make their point by taking the number out of the context of the full experiment. Even given an incredibly small sample, in a type of game often infamous for an aggressive and insulting community, they could only find 11 people making sexist comments. Yet somehow, this is indicative of sexism.

This is not sound science. It hardly qualifies as decent commentary given that is effectively disproves its own point as much as it does exemplify it. And it is a prime example with how we are studying video games wrong. Video games are a fascinating subject and they do have a lot of potential to better understand how people interact with each other. But you can't study that if you don't give gamers and video games the proper respect to study them as you would anything else. And as journalists we have a responsibility to represent science accurately, above all else. Not present which conclusions seem to be best for us and leave out all the important details. There is an opportunity for media to educate the populace on science and the expectations associated with it. Don't waste that on agendas.

Share On:

Featured Video (All Videos)


No author image supplied
Staff Writer

Teacher's aid by day. Gamer by night. And by day, because I play my DS on my lunch break. Ask me about how bad my aim is.