Imagine, if you will, a movie that has a black screen for the entire 90 minutes of its run. How good of a movie would you consider that to be?
Or how about a musical track that had absolutely no sound? Although there is the occasional experimental art piece, such as John Cage's 4'33", for the most part a song with no sound would be considered a very poor song indeed.
In both of these cases, the particular mediums in question have features attributed to that medium. A movie is largely visual and mostly audio with the visual being the important part—a movie could perhaps be made with no sound whatsoever but you wouldn't really be able to have a good movie without any picture whatsoever. There are, of course, silent films but those still make use of a movie's strength as a visual medium. Music is slightly more one dimensional in that it only has sound—it's either there or not there, and so you cannot have what would be considered a good song without any sound.
Video games builds on its predecessors by adding another element of user experience: interactivity. And I would argue that that is the most important part of a game, just like what's shown on screen is the most important part of a movie.
Of course, interactivity also goes hand in hand with feedback. If you push a button and you don't get any sort of result from it I'd argue that that isn't a very interactive medium.
I'm sure there is a video game out there somewhere that deliberately lacks any sort of sound. It can even lack visuals entirely in some cases—take the game Blindside as an example of a completely audio-based video game. But a video game that lacks interactivity would be, in my eyes, a very poor game indeed because it doesn't take advantage of the strengths of the medium. I would say that it wouldn't even qualify as a game unless you use an incredibly broad definition.
And it is the idea of interactivity (and, as a corollary, player agency) that would lead some to declare that Game X or Game Y is "not a game." When they say it's "not a game," they are probably using that as shorthand for "this game has poor interactivity" or "this game has poor player agency."
One of the more classic examples is Dear Esther, a game that is sometimes derisively referred to as a "walking simulator." It has one of the lowest forms of interactivity one could expect; you have to move through the world to advance the story, but that's pretty much it. You don't really have much in the way of change you can affect upon the game world or variety in playthrough. If you were to sit next to someone and watch them play Dear Esther (or a similar game) you would largely have the same experience.
There are similar games with the same sort of problem. I have to wonder if perhaps they wouldn't get as much criticism if they were machinima rather than video games.
Another core example of how a game with poor interactivity can be considered a "bad" game is the idea of Let's Plays. Some people might make the argument that Let's Plays will impact the sales of a game because people will have seen all of the story. If you can get the same experience (or mostly the same experience) of playing a game simply by watching someone else play it, then I would consider it to be a poor game as it doesn't utilize the interactive nature of the medium very well at all.
I'm sure that some people do care solely about the story in a game. However, I feel that this isn't the majority. There's so much that you just don't get to experience by watching someone else play (however you do it). And if you can essentially get the same experience "playing" a highly non-interactive game by simply watching someone else do it on YouTube, then perhaps it is disingenuous to call it a game at all.
That's not to say that a game that is light on interactivity is bad, either. I watched Jesse Cox's playthrough of Beyond Two Souls from start to finish and that is a game with a heavy emphasis on story. I know the vast majority of the story by virtue of having watched the playthrough but there are still different choices I can make and different approaches I can take that will give me something I didn't get simply from watching it.
I like my music to have sound, my movies to have visuals, and my games to have interactivity. Developers will push boundaries, sure, but I feel that if they fail to understand this concept (or deliberately ignore it) they will either earn the ire of the community or something much worse: they'll be wholly ignored.
What's your definition of "game"? What do you feel is absolutely essential for something to be considered a video game? How important is interactivity to you in video games? Let us know in the comments below!