TR Member Perks!

Video games are certainly a fascinating subject aren’t they? As a relatively new phenomenon, they have become the interest of many scientists and researchers. Why not? Video games not only offer a very unique way to study human behavior, but are effective tools in themselves. The same programming used to make video games can be used to make research tools to better understand all kinds of things, from psychology to physics. You would think, given their usefulness, that video games and science would appreciate a friendly relationship, characterized by fairness and mutual respect.

Which makes it all the more insulting when science seems to use its time constantly attempting to sabotage games even if it means sacrificing its integrity in the process. Researchers seem to take abject pleasure in pushing the idea that video games are the absolute worst thing to ever befall society. If you look up studies, you’ll find they tend towards focusing on proving that video games cause everything from violence to sexism to mental disorders. What you’ll also find is the peculiar pattern that these studies seem to assume that going in. That specific journals are prone to never publishing anything that doesn’t completely support that pre-conceived conclusion. And that apparently objectivity takes a back seat to padding your research councils on video games with people who have clear biases towards the “video games cause terrorism” motif.

Indeed it seems if you look outside of the published journals you find a lot of established scientists pointing out that the conclusions peddled by these “studies” aren’t as clear cut as organizations like the American Psychological Association would like you to believe. So why would scientists try so desperately to sabotage an industry that has provided so much to progress and discovery? There are a million conspiracies you could draw from that, but that’s not why we’re here. Most recently, many journalists have been happily pushing a new “study,” which proclaims that players who aren’t as skilled are more likely to be sexist.

A few things to note about this study. First, the skeleton is a direct rip off of this study, conducted by Nicholas Matthews. I say rip off, because despite the fact that very basic procedures are nearly identical—and the behavior being studied very similar—the author doesn’t mention the study once. This could be coincidental, though it should be noted that the author of the original study took a very positive view of gamers in his piece, concluding that aggression is caused not by games themselves but by frustration caused by a lack of ability. It is readily available to anyone with access to basic research libraries and was peer reviewed and released in a fairly reputable journal.

However, even though the basic concepts are the same, that is about all these studies have in common. While the original study was objective, fair, and professional, this study is clearly affected by the biases of the researchers, which makes the highly questionable methods even worse. Rather than creating their own game, allowing the researchers to control the environment, Kasumovic and Kuznekoff used Halo 3. Rather than a carefully designed sample method, they used a sample of convenience based solely on their randomly encountered counterparts. Rather than a consistent measure based not only on observation but self-report, they use a highly unreliable means of determining skill and behavior, one which could be very easily tainted by pre-conceived biases.

You can’t completely blame the researchers though. A large part of the problem with this study is how it has been portrayed in media. That is, as absolute and irrefutable fact. Let’s have a talk about the scientific method, Internet.

First, when conducting an experiment of this kind, you must control for as many factors as possible. This does not simply mean having a control group. For my Senior year of college, I had to conduct a very simple study to practice methods learned in an undergraduate research class. By nature of my status as an undergraduate, the study wasn’t going to be published; was going to have a small, convenience based sample; and likely no one but me, my associate professor, and maybe the professor himself would see it. Even then, I had to control for a lot, and anything I couldn’t control for, I had to explain. Were any of the survey packets different? Was the room too hot? What was I doing? What is the possibility a person’s answer is impacted by their mood at that time? Especially when studying people, there are countless things you have to account for. This study doesn’t account for any of them. It doesn’t even do the basics of tracking demographics of users.

This is why the original study went as far as having their own game designed. When studying video games it may seem like it’s an easy thing to control, but there are more factors as play. Especially if the game is played online. You aren’t witnessing the person in the other room, you can’t see if anything around them is affecting them or their play. You don’t know what their connection is. You can’t even be 100% positive of the gender of the person on the other side.

Second, sample is important. Not just the size, though obviously that is quite important (for the record, despite having such a lazy design, this study has a pitifully small sample). How did you select your sample? The original article used a valid selection process with true randomization. This study used a sample of convenience, usually reserved for students and testing instruments. Essentially, this means the researcher set up shop, then took whoever walked by as their subject. There is no rhyme or reason, and while it may seem random, mathematically and psychologically it isn’t. Halo 3‘s servers aren’t random. They use an algorithm to match players against each other. And this algorithm can be influenced by past games. Essentially, this study is designed so that the actions of past subjects can easily influence the actions of future ones. And it isn’t accounted for by any means.

As a third point, this study is very questionable in its following of informed consent rules. Science, particularly psychology, has not had the most sunny history in terms of participant treatment, so there are many stringent rules in place regarding consent. The researchers actually do address this, stating that consent was not collected because users were anonymous. However verbal consent is allowable and no rules strictly state whether anonymity plays a factor. Technically speaking, all participants should be treated as mostly anonymous. They also state that because users pre-agreed to games being recorded due to accepting the XBox Live Terms of Service, that this counts as exempt under Category 4 of the Institutional Review Board.

But consent means more than just consent to be recorded. Consent requires that the participants know that they are being studied. You do not necessarily have to say why in the beginning—a practice commonly used in experimental studies to avoid participants altering their behavior to fit what researchers expect—but you also have to do a debriefing at the end of the study and give participants full information. None of the justification given covers any of that, because users may agree to being recorded, but that does not mean they agree to being studied. Critically, this article is not only poorly made, but possibly unethical.

And lastly, this one specifically to the media harping this as the absolute truth: science is not a one shot deal. Particularly not psychology. It can take hundreds of studies before scientists can collectively agree that something is essentially fact. One study does not decide human behavior, especially a study as badly constructed as this.

Taking all this into account, it is frightening how seriously people took this study. The paper is littered with statements like these: “This does, however, reinforce the fact that women are entering a very male dominated environment” in inappropriate places like the Results section, typically used only for the dry data, not commentary. Completely ignored by the media is the fact that women are also treated better by high skilled players—apparently by a much larger margin than male players are treated. If you look at the raw numbers, you find this supposedly epidemic problem is represented only by 11 people, and the researchers had to specifically manipulate this just to make their point by taking the number out of the context of the full experiment. Even given an incredibly small sample, in a type of game often infamous for an aggressive and insulting community, they could only find 11 people making sexist comments. Yet somehow, this is indicative of sexism.

This is not sound science. It hardly qualifies as decent commentary given that is effectively disproves its own point as much as it does exemplify it. And it is a prime example with how we are studying video games wrong. Video games are a fascinating subject and they do have a lot of potential to better understand how people interact with each other. But you can’t study that if you don’t give gamers and video games the proper respect to study them as you would anything else. And as journalists we have a responsibility to represent science accurately, above all else. Not present which conclusions seem to be best for us and leave out all the important details. There is an opportunity for media to educate the populace on science and the expectations associated with it. Don’t waste that on agendas.

Kindra Pring

Staff Writer

Teacher's aid by day. Gamer by night. And by day, because I play my DS on my lunch break. Ask me about how bad my aim is.

  • moose

    Its rare that I change my mind on something from an article, not because I am mentally stubborn (sorry I forget the proper word), but because they are usually filled with subjective opinions of an obviously biased author who doesn’t provide adequate sources to back up their assertions, and the articles I read tend to be on subjects I mostly know about beforehand anyway. But this has caused me to change my mind.
    I first heard about this study in the telegraphs (beta) mens section of their website. I completely ignored what the article said and went straight to the link to the study, as the conclusion didn’t match my personal experience, so I wanted to read it for myself. I took a quick read through the entire thing and apart form the typical feminist victim mentality statement in the introduction, the only issue I noticed was that the control group wasn’t included in the data collection or results. But as someone who has an interest in science but isn’t formally educated beyond mandatory schooling, whilst I wasn’t going to form a conclusive opinion based on a single study, I didn’t see any reason not to trust the results.
    After reading this article I am kicking myself over not spotting issues which I admit I would have thought “well duh” if I had read this article first and then looked at the study for myself. From 12 years old I was taught that the only variable should be the thing you are trying to study or something along those lines, so for that alone I direct this facepalm towards myself:

  • Screech Screecher

    Most studies these day seem to start with the answer that provides the most funding and then work backwards to provide a suitable experiment.

  • Cred

    people just see the word “Study” but don’t care about the quality or the bias in it
    wise up people

  • NorBdelta

    Annoyingly the article negates the fact that there was substantial positive comments by higher skilled players. It just shows that some people are essentially sore losers and probably pick on whom-so-ever is speaking at the time.

    This study needs to be repeated with greater controls in place. I suspect that just by speaking the player became a venting target from so-called losing players.

  • Cred

    but if they tried exploring multiple possibilities to figure out what makes sense and if they tried to employ methods that didn’t lead to biased results how could they push the anti gamer narrative they’re being sponsored to provide ?

  • cptk

    “This study needs to be repeated with greater controls in place. I
    suspect that just by speaking the player became a venting target from
    so-called losing players.”

    This is a key point IMO.

    Purely anecdotal but in my experiance whether it’s recreation (sport, video/board/card games, etc ) or not (career, possesions, love-life, etc) those who are doing well tend to be politer to those who are not doing well and those who aren’t doing well will react badly by any real or imagined boasting. AKA ‘haters gonna hate’ for the post-music MTV generation.

    Any study targeting video games would need to be compared with similiar controlled studies on other activities in order to determine whether there is a significant difference to basic human behaviour.
    Additionally, there needs to be a control on the lack of physical proximity because there is almost certainly a human trait to be more aggressive when not dealing with someone face to face. We’ve seen that in motorists for decades.

    In fact, just spending 10 minutes thinking about how one would conduct the study provides much better considerations than those made in the article itself.

  • Unsafe Ideas

    This was not study on videogames nor gamers, this study tested general male-status and vs behavior psychological hypothesis on Halo 3 players. That whole gaming world is now trying to make gaming conclusions out of that is not fault of the study, but rather fault of the gaming world.

    This study is doing exactly what you demand when you ask for “more experiments”. It is adding one more (small) experiment to set of studies about behavior intersexual behavior. It is just doing it in the context of games. It does not study games themselves and it does not care about whether gamers display more sexist behavior then Wall Street execs or nuns.

    Study is a rip-off of another study is the most ridiculous complaint ever. Experiments like this are not supposed to be super original, they are supposed to be repeated in many different domains under slightly different conditions until we have many of them. The “references list omits Nicholas Matthews findings in another study” would make some sense, through I do not know how important Nicholas Matthews or whether they both “ripped off” a common study.

    Also, self reporting is not superior to third party observations, studies based on self reporting are usually criticized for being that way. Self reporting is notoriously unreliable.

    If the study would be testing things on made up game, we would complain that it says nothing about real games because made up game is nothing like a real one. There would be no such thing as status in that newly made up game nor real pressure to win (which online game has) and fear of loosing would be lower. Subjects would know they are being observed and they would change their behavior.

  • Kain Yusanagi

    Wen they basically plagarise without credit, that’s not a ridiculous complaint.

  • Unsafe Ideas

    If they would plagiarize indeed yes. Two studies using the same “basic procedures” is not plagiarizing. The two studies have even two different goals.

    This one wanted to look for relationship between status and behavior. It is also trying to decide between social constructionist and evolutionary explanations for sexist behavior – the difference is described in text. It did not cared about games themselves nor made conclusions about games/gamers – the tested hypothesis was general and game was used as a place where to measure it.

    The other one is trying to find out effect of in-game skill on hostility, aggression and perception of violence in game and during the game. It did cared about video game itself, game itself was subject of the study. On the other hand, it did not even cared about gender relations one bit.

    This study asks: when sexist behavior happens, is it because cultural factors (women do not belong here, I think it is ok to act like that) or is it rather result of competition for status (my status is under greater threat and I lash out to regain it)? The other study ask: does your skill affect your hostility and perception of violence in games?

    Neither of those studies makes any conclusion about gamers in general being more or less sexist then the rest of the society, males being worst then women in general (do romance fans have similar studies in opposite?).

  • Mark Samenfink

    Feel you on this, I’ve been trying to demonstrate all the flaws in this study on twitter for days, every time I try I get ridiculous responses, immediate insults, or blocked.

  • KindraPring

    This certainly was a study on video games and gamers, regardless of how you would like to twist the results to match a broad anthropological point. It was done on an online server, with people who are primarily gamers. Saying “this is not about games” would be like studying the cardiovascular capabilities of a basketball player and then saying the study wasn’t about basketball players. You may not have wanted that, but that is what it is. The commentary throughout seems to indicate that clearly, this was as much about video games as anything else.

    I never demanded that actually. I demanded a lot of things, but that was not one of them, at least not of the researchers. I demanded that of the media who tends to present single studies as the end all be all. And again, the commentary in the study would suggest otherwise – if this has nothing to do with gamers, then why bring up the “Women in gaming have it so hard” rhetoric? In the results section no less?

    Never said they were. But you still cite it. It’s odd that despite the similar methods they never once came across this study. And yes, they are similar enough. In fact when I first heard about this study, I thought people were talking about the previous study at first glance and just misinterpreting the results.

    I never said self reporting is superior. I said it helps, especially in situations like these, and adds further context to what is going on.

    No, we wouldn’t. When I wrote about the previous study, most people thought their making of their own game was pretty cool and some of them even said they’d like to play it to get a better look at what they specifically made.

    Your comment is full of assumptions and downright falsehoods, like claiming I’ve said things I never did.

  • Niwjere
  • grumpus

    Well, it is a ridiculous complaint when the author of this piece declares that the study in question is a ‘direct rip-off’ of another study from 2015, while a cursory reading of the Kasumovic paper would show the following:

    “Data collection involved analyzing video recordings generated from Kuznekoff and Rose’s (2013) original study.”

    The rest of this article is just as ridiculous, and does a good job of showing that the author doesn’t know anything about scientific research, statistical analysis, or anything else that might help when trying to perform a critical analysis of a journal article.

  • CanofSand

    I honestly feel the world would be a better place (not perfect, not great, but better) if Twitter was never invented.