Analysis of steam user reviews

Posted in

#1 by bunnyadvocate
2019-03-11 at 16:04
If the quote “our language is the reflection of ourselves” is true, then what could we learn from Steam user reviews (other than that we deserve every bad game ever published there)? I thought it’d be fun to generate some word clouds of Steam VN user reviews: what words tend to pop up in positive reviews vs negative ones, which words are overused in 18+ VNs, and how old can I make myself feel by seeing all these memes references.

I'd recommend reading this analysis post over here because I can embed images in the post there which makes it easier to read.

Data collection

Using the Steam API, I collected the English user reviews for any VN released before 2019 on Steam, giving us 250,000 reviews in total. I filtered out Doki Doki Literature Club’s 71,000 reviews to prevent the results being skewed too heavily towards one VN. I also filtered out any word that appeared in fewer than 5 different VNs to prevent character names from popular VNs from filling up the results too much.

Word associations

A higher resolution set of these word-cloud images can be found here. The larger a word is the more frequently it’s used.

link

First up is a comparison of what words tend to appear more often in VN reviews vs a random selection of other games on Steam. There’s nothing too surprising here, it’s mostly just popular VN series and various “weeaboo” terms. “Uncensored” also pops up quite prominently, showing how much 18+ content matters to the medium.

link

Next is the list of words that are more likely to come up in reviews the user submitted as a thumbs-up (a positive review). While there’s all the usual flattering terms you’d expect, it’s fun to see how many words traditionally associated with negative feelings come up here: tears, cry, sadness, and bittersweet all show up indicating how much we seek a deep emotional release in VNs... also “memes.”

link

What goes up must come down, here we have words that tend to come up in thumbs-down Steam reviews. Despite VNs being a visual medium, almost all the terms would seem to describe their opinion on the story/writing rather than the visuals, with “boring,” “waste,” and “stupid” all coming up more often than “ugly.” This suggests the most common way a VN can disappoint its users ie through its writing rather than its aesthetic, although this may because it’s easier to judge the visuals from screenshots before you purchase the VN.

link

The English-developed VNs reviews are mostly full of titles of popular EVNs, but we can also see how much more socially progressive EVNs are, with terms like gender, diversity, and lgbt coming up more often than in JVNs.

link

Truly Japan is the land of wonder, or at least kawaii catgirls according to these results. It’s curious that “fighting” would be linked to JVNs, perhaps because there’s so few EVNs that depict violence.

link

Thank you Steam reviews for telling us in the review that a free VN is free. Truly a valuable service.

link

There’s a certain irony that the words that come up more often in commercial VNs are all about how to get it cheaper: sale, bundle, and discount are all prominent terms.

link

These are the words that tend to come up more often in otome VNs reviews (VNs with a female protagonist pursuing male love interests). Words associated with fantasy type settings pop up frequently, queen, princess, and prince are all present which is unsurprising given otome’s love of the fantasy genre.

link

These are the cursed words which come up more often in 18+ VNs, what is lacking in eloquence is at least made up in “tiddies.”

link

These are the words that tend to come up in shorter reviews. We’re told a picture paints a thousand words, but who needs a thousand words when you have swear words to convey meaning instead? Who needs verbose reviews when you have “fuck gay catgirl boobs.” I’m sold on that VN already.

You can find higher resolution versions of all these word cloud images here (as well as the longer-review wordcloud which wasn’t included here because it’s boring). You can also view the results in spreadsheet format here, which is useful for reading the smaller (less-common) words.

Review trends

I’d be ashamed to put my name to an analysis post that didn’t have a graph or two, so to cleanse the palette after all those word clouds here are a few graphs looking at some review statistics.

link

Typically reviews tend to be fairly short, just a few sentences at 150 characters, but there’s no stopping the more long-winded of us who might be few in number, but dramatically pull up the average wordcount with their detailed account of everything that happens in a VN.

link

I expected that the longer a review was, the more likely it would be that the reviewer liked it. It’s the fans who want to talk about it the most isn’t it? Well it turns out to be the other way around, the longer a review is, the more likely it is that the reviewer disliked it. Perhaps unhappy user’s feel the need to justify their opinion more when it goes against the general positive consensus on Steam, where 86% of all reviews are positive. So in accordance with this trend and the excessive length of this post, I hereby give my Steam VN review analysis a thumbs down. Not enough kawaii catgirls.

While I don’t think any of these results were surprising, I hope you found it a fun read. If you’re interested in more, check out my other posts on tumblr. Feel free to give any feedback or suggestions for other analyses either here, twitter, or Discord (Sunleaf_Willow /(^ n ^=)\#1616).

Thanks to Lunaterra, /u/8cccc9, and Elm for their input during this analysis.Last modified on 2019-03-11 at 16:06
#2 by erohatasensei
2019-03-11 at 23:59
Another interesting question. How many steam reviews are bought and/or abuse the fact that the developers give out easy copies for subpar/undetailed reviews?

I know that in the chinese community like 90% of reviews are bought or encouraged by raffle entries so it fucks up the whole system.
#3 by bunnyadvocate
2019-03-12 at 09:25
#2 That's a really good question, I was curious about that too. There's a flag for whether someone purchased a game on Steam or just entered a key, so I tried seeing what words were more common in those reviews for commercial VNs and got this: link

It most looks like folk talking about getting it via kickstarter rewards or bundles.
#4 by encrypted12345
2019-03-12 at 16:43
@2 I think if you're caught buying reviews, Steam will ban you from the store entirely. It probably does happen on the English side, but probably not 90% at the very least.
#5 by lunaterra
2019-03-12 at 20:24
It used to happen a lot, but a couple things happened to make it less common (I won't go so far as to say that it doesn't happen anymore, considering that games get taken down for review manipulation on a regular basis):

1. Steam no longer counts key activations in the review score. If you want your review to count, you have to buy the game via Steam itself.

2. Gleam.io, the giveaway site that many of these devs used to give out keys, no longer allows mass game key giveaways.

There's still ways to get past these, but it's more difficult for less reward.

Reply

You must be logged in to reply to this thread.