Can we fix the Bayesian Rating Formula?

Posted in

#1 by animecac
2022-05-07 at 10:27
< report >Hi all,

I'm sure I'm not the first to post this but couldn't find any discussion after a quick search and felt it could use some addressing at this point in the database's history.

I feel we could use a better formula on the Bayesian rating that isn't quite so harsh on newer or less popular titles. I understand we do need it adjusted so we don't get random new titles spammed to the top of the list. At the same time, the list is so dominated by titles that were released at the apex of visual novel development that it will likely be impossible to top them. Even if a new release were hypothetically as good or better than these titles, and scored as such, it'd be unlikely to get near the top due to the popularity issue.

To me this breaks the ability for the database to be more representative and help newer titles find more players. It also creates an undue implicit bias to older titles. I understand why this is the case, but to me this helps reinforce the idea that the heyday of visual novels has passed.

I understand that you can use your own search fields to bypass the Bayesian ratings, but realistically only some will do this. I would love to more easily find newer titles that are really good. In the event they aren't as good as they appear and climb higher than they ought in the short term, this will only encourage more people to play the game (which is a good thing) and re-correct the scores back to where they should be. I understand that newer titles may climb up the list a bit, but it won't be long before they go back down. Again, I think this is healthy and keeps some sense of freshness in the list.

Finally, I'm not saying there should be no Bayesian formula. Just that it is my opinion that it could be done in such a way to not so heavily penalize newer or less popular titles, although I understand it needs to do it to some degree.

What do other people think? Am I alone in thinking this?Last modified on 2022-05-07 at 10:35
#2 by bijak
2022-05-07 at 10:43
< report >I agree with this. I looked at the source code and it seems to be using the average rating and average number of votes on entries to determine the prior. Agree with the average rating being used - but using the average number of votes is way stronger than most priors I've ever seen used. Most of the time I see something like the first quartile or lessLast modified on 2022-05-07 at 10:43
#3 by Ezezin
2022-05-07 at 11:43
< report >There was a suggestion on how ratings can be calculated in t17035, but it ended in... I actually don't understand a thing on that thread.Last modified on 2022-05-07 at 11:45
#4 by animecac
2022-05-07 at 11:54
< report >@2 I agree. I just think tweaking the amount of weighting given to the number of votes could do it. If that doesn't work perhaps an alternative score predictor based on voting patterns / other parameters to add to the weighting if it has less votes? I don't know enough in this space but I do know there are some great formulas already out there to use.

@3 I just read it, I don't understand much either! From what I understand given what was said on post 7, to quote him "What I'm suggesting wouldn't change the rankings of anything at all". So I don't think that thread was about changing the rankings and Bayesian Formula, but rather adding new functionality to the search functions. At least I think... haha.Last modified on 2022-05-07 at 11:57
#5 by ashitaka
2022-05-09 at 00:33
< report >I also agree with this. The current raiting favors the most popular older entries too much. There's not a single VN from 2021 and 2022 in the top 100, and only 3 from 2018 to 2020 in the top 50.

"The Sekimeiya: Spun Glass" link has a rating of 8.22 but is only at #226. Taking the actual ratings of all VNs with more than 100 votes, for example, it would very likely be in the top 100 and get a bit more exposure this way. (I haven't played it yet, so I have no agenda regarding it.)

Punishment for low amount of votes should go down more and more after around 10 votes and there should be almost none when reaching 100 votes. "Sekimeiya" has 132 votes and its rating goes down from 8.22 to 7.63 with the current algorithm. That's way too much. It should be something like 8.15.
#6 by Mrkew
2022-05-09 at 01:06
< report >
The current raiting favors the most popular older entries too much. There's not a single VN from 2021 and 2022 in the top 100
Correction, the rating favors translated VNs. While it's technically true that there's not a single one from 2021/2022, some of those titles in top 100 got there only because a recent translation upped the vote count dramatically.
#7 by animecac
2022-05-09 at 01:55
< report >@5 Completely agree. I personally think Sekimeiya is over rated, but enough people enjoy it so it deserves to go way up the list. If I'm wrong and it isn't overrated, more will play it and it'll be vindicated with more scores to validate it. If I'm right and it is overrated, the extra exposure will help more people play it and balance the score.

In either case the result is good, and for now it deserves to go way up the list as enough people have scored it highly. I can't help but think this type of thing is healthy and will add a much needed fresh wind into the databse and community as a whole to see new titles appear in the top 100 more regularly.Last modified on 2022-05-09 at 01:57
#8 by Ileca
2022-05-09 at 04:43
< report >What prevents you guys from ordering VN by vote average, number of votes or popularity score? There isn't one top 50.
You can also use vote average and set a minimum threshold of votes that fits you.
I use popularity score and only to get a general idea on what to prioritize in my wishlist.

The Sekimeiya: Spun Glass
Vote average: #863
Bayesian rating: #226
Popularity score: #1729
Vote average with 300 votes minimum: not enough votes

Bayesian rating is ironically the only way to discover Sekimeiya.

With a threshold of 100, it is #91 but the top is quite a mess. You know what is #208, above Katawa Shoujo? Lessons in Love XDD

I do agree though that Bayesian rating should be dropped past a certain number of votes, like 300 (involves 686 VN). That's what I thought when I still cared about tops. Now, I couldn't care less.
#9 by animecac
2022-05-09 at 08:33
< report >@8 Nothing stops us using the search features, but my initial post addressed that line of thinking and why I think despite those search features it is still worth tweaking the current Beyasian system. It boils down to the fact that whilst on an individual level we can look past the current filtering, there is no doubt that it currently reinforces certain ideas and makes certain titles harder to find, especially for those who use this database on a superficial level and never get familiar with those functions, which I'd guess is a large proportion of users.

I only see positives by updating the formula and think it could bring a new life to the database. Personally I agree with you in the sense that I don't put too much stock on the placing of the list, however at the same time I recognise many do and the perception of a rating score, and the ease of accessibility of certain titles strongly influences the community whether we want it to or not.

For example, it took me a long time to find Sekimeiya despite my own search filters and scrolling through many pages. I don't care whether it is rated 10th or 150th in principle, but I do in terms of finding new titles that may be worth checking out.Last modified on 2022-05-09 at 08:41
#10 by Yorhel
2022-05-09 at 13:35
< report >Not sure I fully agree with the premise, the top 50 seems pretty fluid as new translations get released. Muramasa and WA2 went up super fast after their English TL, so it seems to be working fine to me.

using the average number of votes is way stronger than most priors I've ever seen used. Most of the time I see something like the first quartile or less
The currently used average is 55. The first quartile is... 2. That seems a bit extreme in the opposite end. Even the median seems fairly low at 7. How do people typically select this parameter?

I'm open to reasonable changes to the rating algorithm, but keep in mind that I'll be somewhat conservative. There's no such thing as an algorithm and parameters that everyone is going to agree on and too many people are taking the rankings far too seriously, so changes have a good chance to incite public outrage.

I [..] think it could bring a new life to the database
This just triggers me on so many levels. :)
#11 by animecac
2022-05-09 at 20:55
< report >@10

I'm open to reasonable changes to the rating algorithm, but keep in mind that I'll be somewhat conservative.

Thanks, I'll look into it more. I agree with being conservative. I don't think it needs a large tweak by any means personally.

Not sure I fully agree with the premise, the top 50 seems pretty fluid as new translations get released. Muramasa and WA2 went up super fast after their English TL, so it seems to be working fine to me.

I think both Muramasa and WA2 are unique because they aren't new titles and already have a huge following, because they are so often called the last of the Kamige and have had well over 10 years of build up waiting for a translation. They already had some popularity going in and then a huge spike due to that build up. Outside of a scenario like that, new titles almost never crack into the top.

This just triggers me on so many levels. :)

Sorry I didn't intend to trigger you! Nor did that statement intend imply their isn't life. More so just that some more movement in the list itself, and having it slightly more adaptable to truly new titles will imo keep it feeling more dynamic.Last modified on 2022-05-09 at 21:00
#12 by Ileca
2022-05-09 at 20:58
< report >I propose a new default: mod curated ordering. MLA locked on #1, #2 to #49 are PC98 games, #50 to #99 3D Patreon games.
#13 by 4digitmen
2022-05-09 at 23:10
< report >The problem with discoverability isn't that the Bayesian rating isn't working, it's that there isn't a system in place for recommendations. I'm sure that Sekimeiya would be easier to discover if it was recommended on the Infinity series VN pages, and not only that it would be recommended to people who are more likely to enjoy it. Messing with the top 50 or 100 or 200 or beyond won't really change much, because really, it doesn't mean a thing. The top VN on the website says enough about how little it matters.

Edit: This was already discussed in another thread, and I think Yorhel was in the replies there too, so I at least hope it's already being considered.Last modified on 2022-05-09 at 23:14
#14 by Arsym
2022-05-11 at 21:54
< report >@13 is right. Regardless of the rating system, The Sekimeiya will never be a top 50 title for most people. It simply doesn't have enough mass appeal. Niches exist, specific tastes exist. The solution to discoverability isn't shuffling a title from page 13 to page 4 of the global rankings. The solutions is to have a larger viewing of games that are relevant to your interests.

The problem with general ratings is they wipe out nuance. I don't trust people who mainly enjoy death games to recommend a good nukige title. I don't have a problem if people who enjoy political intrigue think Utawarerumono: Futari no Hakuoro is a 10 as long as I can see what readers of specific kinds of mystery enjoy.

Good recommendations are the answer. That's a whole discussion in itself, but VNs can be recommended on other VN pages directly, or users can put VNs in "collections" that people can browse. The whole thing should probably be driven by votes, but you can also use existing data to create priors.
#15 by Ileca
2022-05-11 at 22:43
< report >You want a good death game? Check the Death Game tag, duh. You don't need a new feature for that. It's like you are saying that the only way to search for stuff is using the unfiltered ranking, which is stupid at best.
Sekimeiya is rated incredibly high. It will eventually appear in one of your search if you are dedicated enough. I did a quick try with language:English + tag:Mystery + number of votes:>=50, ordered by vote average and got Sekimeiya in the first 50 results. With bayesian rating, it is #101. With a less general tag such as Confinement, it's #16 by default.
Maybe stop relying on general ranking, and start exploring the db properly.

Recommendations will not give you necessarily good VN or make VN more discoverable. From a VN page, you will be supposed to either have similar VN (not good, similar) or people who enjoyed this VN also enjoyed those (and get popular VN everywhere).
#16 by Arsym
2022-05-11 at 23:22
< report >
You want a good death game? Check the Death Game tag, duh.

Not that easy. Using the Mystery tag as an example, the type and concentration of mystery present in Muv-Luv Alternative, STEINS;GATE and Umineko no Naku Koro ni are vastly different. Depending on which one you enjoy, I can give entirely different recommendations. You also get outliers like CLANNAD and Ao no Kanata no Four Rhythm in there. As powerful as the search is, it's not all-mighty, nor is it always easy to find what you want.

There might be a case for scoring titles based on tag similarity. I dunno, all I'm saying is that changing the ranking ain't it.
#17 by 4digitmen
2022-05-11 at 23:41
< report >#15
I know you detest being professional, but the passive-aggressive response here is unnecessary and not productive. You not only started by directly saying the suggestion is stupid, but you then basically told him that what he's doing is wrong and that he should learn the complicated tag system like you know it.

Instead of assuming how the recommendations would work, you could have just looked toward other weeb media websites like MAL or Anilist which have both recommendations and tags. I have no idea why you're so insistent on how the tag system must exist in isolation and the recommendation system is simply useless, when your direct compadres are using both to successful results.

I'd also like to point that out of these 3 (vndb, Anilist, MAL) Anilist has a nearly identical tag system, although with a much more streamlined and easier to use search system for tags. It's arrogant to assume that every user on vndb knows the in-and-outs of the tag system, or that they're willing to learn it in its current state.Last modified on 2022-05-11 at 23:41
#18 by Ileca
2022-05-12 at 01:39
< report >Arsym, I am not against a recommendation system, just against this jump from the general vote rankings obsession to recommendations as The Answer, without considering that the current search system is powerful enough to get you where you want to go 3/4 of the time. I only replied to you because of how you framed recos as being the only solution to discoverability, like if we couldn't search anything outside of rankings. I admit I was a little rude to you and I apologize for that.

Btw, the "type and concentration of mystery present" should be handled by coefficient, tho it is not well used with lazy users not touching the default value, with coef 2 being flooded with pretty much everything. Honestly, the coef should have no default.

In fact, the vndb's tag system is so powerful compared to other websites that end up relying on user recos instead, that if reco system there should be, I think it should rely on tag cloud's similarity without caring about votes (tho you could order the result this way). It would go much further than your typical "page has two, three measly user recos" or "page has a bunch of popular VN who-is-not-a-noob already knows about, with plenty of random stuff inbetween". More VN have a fair number of tags compared to their number of votes. You just need one guy to fill tags, after all. That would make for interesting recommendations imo.

Anyway, we agree that the general rankings are for quick and dirty ways to know what's popular when you start reading VN. They are not the solution for discoverability.
#19 by Arsym
2022-05-12 at 03:53
< report >Don't worry about it. I should have said "an answer" rather than "the answer".

I wonder how far the tags can be pushed. I'll need to experiment with them and see what kind of ranking/similarity metrics can be devised.

But yeah, leave Bayesian alone, leave average scores alone. They're not perfect and tweaking them won't eliminate their key faults. Make something better.
#20 by animecac
2022-05-12 at 10:45
< report >@14 I don't think recommendations strikes the heart of the problems I'm putting forward, as good as that feature might be.

I think when you have a formula for a rating system that is so heavily tilted towards popular titles, and it fails to be representative on the quality of newer titles, you have a problem. It then becomes another type of 'popular' tag in some way. This is evident by the fact as pointed out earlier in this thread that all the top visual novels are old titles.

The ability to find newer titles of quality will be a great by-product of fixing this but is not the core of the issue, which is why I don't think recommendations will fix it (at least not directly). The Sekimeiya thing was just an example but I'm not hung up about specific titles, more about the general trend of the impossibility of newer titles climbing a rating list (note - we already have a popularity tag so I don't think the rating tag needs to be so heavily weighted to popular titles).

So whilst the list needs some weighting towards the more popular titles to ensure the accuracy and confidence of the scores, the main point of this thread is about discussing the correct balance of that weighting. Not specifically about finding new titles.Last modified on 2022-05-12 at 10:53
#21 by dchsflii
2022-05-18 at 04:37
< report >@3 @4 The discussion in t17035 is about a separate idea.

Changing the details of the prior for the Bayesian mean formula could be interesting. There are heuristics, but ultimately the choice of exactly what to use is somewhat subjective. I think anyone can download the vote data, so if you wanted you could do that, try out different choices, see what feels right to you, and present your results. That might be more convincing than a general proposal where the details are vague and give people a sense of whether this proposed change would actually be something they liked.Last modified on 2022-05-18 at 04:44
#22 by animecac
2022-05-18 at 09:22
< report >@21 That's a great point. If you or anyone else reading this is great with this stuff and would like to input let me know. I can figure it out but may take me a bit of time.
#23 by schlaefer
2022-05-18 at 11:06
< report >@10
Using the average vote count seems a bit excessive. It would need a considerable amount of votes to make a meaningful change to the rating. Half the average might work better.

Or another idea: Use the difference to fill up the vote count until the average is reached. So if avg vote count is 55 and real vote count for a specific title is 5, add 50 votes with an average rating. If real vote count is 30 add 25 more and so on. If there are 55 or more votes don't add any.
That should help semi-popular titles. And 55 votes seems like a lot, so it should be enough to calculate an accurate average on it's own.

Reply

You must be logged in to reply to this thread.