Weighted voting system

Posted in

#1 by space-ranger
2012-05-21 at 21:07
I looked at a trolling discussion and started wondering how troll votes could be filtered out automatically without hurting genuine voters and I came up with using weighted votes.

Concept:
Each user needs a troll score. This score will be the average of how different the votes are from 6.
Example:
One user as voted (5,6,7,8).
This gives the scores (1,0,1,2) and the average is 1. This means this guy's troll score is 1.

The weight to the votes should then be:
Scores below 2: 1
Scores between 2 and 4: dropping linear from 1 to 0.
More than 4: 0

Why did I pick those numbers?
THe answer is simple: I tested several values and those given here provides the wanted result.
The majority of the randomly selected users is between 1,5 and 2
The majority of those above 2 gets a weight 0,95+
Lowest score found is 1,449, which happened to be Yorhel (one of the few not so random picks)

Lowest score I encountered was 2,5, which gave a weight of 0,75. This happened because precisely half of the votes were either 9 or 10. I get the impression that this wasn't the intended use of votes, which means suppressing such scores with a lower weight would be a nice bonus. I don't think he was trolling though.

I constructed trolls, which gave 10s, 1s and so on and they all ended up with a weight of 0, meaning they will not affect the average votes calculation of the attacked VNs.

The math:

Score calculation:
score=ABS(6-X1)+ABS(6-X2)+...+ABS(6-Xn) divided by n. X is the actual vote and n is the number of votes.

If score is less than 2 or more than 4, set weight to 1 or 0 respectively, otherwise
Weight=1-(score-2)/2

Note: I considered using a real statistical approach with standard deviation and such, but quickly discarded the idea because it would be too CPU intensive.


So what do you think? Would this be a decent way to a decent way to avoid troll and get people to use vote values as intended?
#2 by yorhel
2012-05-21 at 21:16
Each user needs a troll score. This score will be the average of how different the votes are from 6.
Oh please, no. I disagree with the entire notion that we should somehow penalize users that happen to have different voting patterns than what is considered "average". So this idea won't happen. For the weighted average, I'd suspect t1464 will do well enough to filter out so-called "trolls". The current popularity rankings already are much less affected by "trolls" because it also takes the users' vote count and distribution in account (whether or not it is in any way according to some norm doesn't matter here, but it does tend to favour sensible voters).Last modified on 2012-05-21 at 21:20
#3 by space-ranger
2012-05-21 at 21:31
The current popularity rankings already are much less affected by "trolls" because it also takes the users' vote count and distribution in account
Oh good. It's actually more advanced than I thought. Then the problem is not as severe as I thought.

As for what would be "average" in the system described here would actually span from average of votes going from 4 to 8, which does gives quite a lot of freedom. Doesn't matter though since the whole idea is discarded. Truncated average could likely do better overall, though it would be much harder to get 10 votes to count.
#4 by ganchan
2012-05-21 at 21:46
Well, I like the idea, only that I don't agree that they affect to the votes per se, only to help us to find trolls.
#5 by space-ranger
2012-05-21 at 21:56
Well, I like the idea, only that I don't agree that they affect to the votes per se, only to help us to find trolls.
I suppose it could be used to sound some internal alarm. Considering placing half the votes to 9 or 10 gave a score of 2,5, I say a score of 3 or higher validates a check for trolling. All 10s gives 4 and all 1s gives 5 (which is max score). The problem with that approach is that it means somebody should actually manually check each person. How much time will that take? And how many people will actually have to be checked?Last modified on 2012-05-21 at 22:06
#6 by warfoki
2012-05-21 at 22:57
Speaking of troll votes, check this guy out: u26894

I know, this is not a report thread but I'm too lazy to find a more appropriate one.
#7 by soketsu
2012-05-22 at 01:56
I have my own idea for weighted average voting system but I didn't even bother to air it to yorhel for some reasons... But now that there is this thread, I guess it time for it. It's a little bit different from truncation or particular-vote-referenced mechanics as what ranger-san suggested. It's about the number of votes of the voter.

Trolls and casual readers usually cast only few votes. Veteran VN readers, on the other hand, usually have long vote lists. IMO, the more VNs you read = the more polished your rating standards = the more reliable your votes become = the more valuable your vote list for the database... and it would only be evident in your account's long vote list. You see, loss' 10 would be far more sensible than pendelhaven's 10 (sorry for being Judas, kapateed... yung concert kase ni Gaga eh... :P). ...So I think a votecount-referenced weight would be a nice idea.

_________________________________________________
Let x=vote-count for one user account, v=a vote for the VN, m=average(mean) VN rating, w=weight, n=total number of counted votes for the VN

Weights(?):
_____ 0<x<10 : votes will only be displayed in the user account but will not be displayed in the respective VN profile and counts nothing in computing for the average for each respective VNs

_____10<x<20 : votes will be displayed in both the user account and the respective VN profile but still counts nothing in computing for the average for each respective VNs

_____20<x<40 : each vote counts with a weight of 0.5 and will be displayed both in the VN profile and the user account

____40<x<80 : each vote counts a weight of 1 and will be displayed both in the VN profile and the user account

____80<x<160 : each vote counts a weight of 1.5 and will be displayed both in the VN profile and the user account

____160<x<320 : each vote counts a weight of 2 and will be displayed both in the VN profile and the user account

____x>=320 : each vote counts a weight of 3 and will be displayed both in the VN profile and the user account

The conventional average formula would be good enough: m=∑(v*w)/n
_________________________________________________

With that system, noob readers and trolls won't really have much significant influence in the average ratings. On the other hand, experienced readers be given more considerations and their efforts in casting votes will be rewarded.
Bad things are, it is a biased system (truncated average is more biased, IMO) and not totally troll-proof. But I guess it is enough to deter most of 'em with only the few diligent ones persist.Last modified on 2012-05-22 at 03:12
#8 by loctar87
2012-05-22 at 02:54
@7 One of the problems with voting systems is that you always need to consider human nature. If my votes don't count for much unless I have lots of votes, then an obvious reaction is to make lots of fake votes on unimportant VNs. Sure, it would take more effort than some trolls are willing to put in, but the people making fake votes might not only be trolls under that system. Average people who want their vote to be heard might be tempted to do an obvious exploit like that.
#9 by soketsu
2012-05-22 at 03:08
...that's why I said it's biased... and it also creates more "walls and ladders" than what we already have in the current system.
#10 by thajunk
2012-05-22 at 06:59
all these troll arguments can be easily rectified if yorhel just implements one of the systems secretly

then no troll can counter it since they wouldnt know what theyre up against anyway
#11 by overmage
2012-05-22 at 10:36
The current system already uses a bayesian average...
#12 by space-ranger
2012-05-22 at 12:52
Speaking of troll votes, check this guy out: u26894
He would be given a troll score of 4, which would give a weight of 0 in the system I proposed. Makes me wonder if it would work just fine if it would allow a troll score of 3 or 3,5, but didn't count any higher scores. Still "just" creating an alert event that the new user looks like a troll based on score wouldn't be bad.

It would be interesting to make a list of users with a troll score more than say 3 or 3,5 to see if it makes even a single false positive.

I wondered about the same approach as soketsu, though I discarded it because it would make trolls work harder and it would prevent genuine users from start voting. 40 votes appears to be a lot if you only tried a few titles while the 50 votes on 1 troll wouldn't have a problem.

all these troll arguments can be easily rectified if yorhel just implements one of the systems secretly

then no troll can counter it since they wouldnt know what theyre up against anyway
Agreed, though the majority of trolls would most likely not study the mechanics of the attacked system and would just go "cool, I can make 50 accounts giving 10 to this poor title".
#13 by vethalon
2012-06-04 at 15:17
The big problem is that numbers really don't work well to represent anything. Normally the average vote for a solid, working and interesting vn should be around 5 or 6 which is, in the eyes of most people, crap. We can see the same thing with testscores for videogames, anything below 80% isn't even worth playing, that's stupid.

Voting with scores from 1 to 10 are to subjective, let's say someone doesn't really like VN's but hasn't realized it jet, he would probably rate everything quite low.

The problem I had was that the first VN I ever played was Katawa Shoujo, which totally blew me away, never had any Videogame touched me so deep. I spent about 60h to get everything out of it and still weeks later I couldn't think about much else. So I came here and voted it 10 out of 10. I've been searching for something that comes close ever since(sugestions are highly appretiated ;-) ) and realized that there are many great VN's, so I set my vote down to 9.

It is really hard to make a working voting system, especially for something as subjective as art.

The best thing I feel would be to make it necessary to write at least a couple of words for each vote you make. If you give a Visual Novel a score you should at least have spent a couple of hours with it (optimally reached at least one of the endings) so it shouldn't be to much to ask for a little statement.
#14Post deleted.
#15 by space-ranger
2012-06-04 at 17:49
Normally the average vote for a solid, working and interesting vn should be around 5 or 6 which is, in the eyes of most people, crap
This is my initial impression too. Quote a number of people gives 5 as the lowest score and 8 being the most common one. This is why I came up with a system, which would make people count less if they fail to use the scale as intended. I understand people who picks titles rated quite high by others and avoids low rated ones. However I have seen people who used 9 and 10 for half their votes. When voting 1 or 10, a message appears asking if you are sure because it's a rare rating.

The idea is discarded though and I can't see a way it would ever be implemented.
#16 by horseband
2012-06-04 at 22:04
Normally the average vote for a solid, working and interesting vn should be around 5 or 6 which is, in the eyes of most people, crap

Each score has a descriptor next to it on the voting pull down. A 4 is "weak", 5 is "so-so", 6 is "average". From what I've seen most people seem to vote in line with these descriptors and most games seem to fit well within these descriptors. According to the website a "solid, working and interesting VN" is not a 5. "so-so" is nowhere near "A solid, working and interesting VN".

The voting range is a lot better than with video games. With video games many people review with the idea that anything below an 80% is bad. That's definitively not the case on here. When you have a decent amount of votes the extreme voters should have minimal impact and cross each other out.

Are there really that many Vns that have scores you don't agree with? I really haven't come across a VN that had a decent amount of votes and said, "Holy crap how did it get that score?"

However I have seen people who used 9 and 10 for half their votes. When voting 1 or 10, a message appears asking if you are sure because it's a rare rating.

If they legitimately thought those games were masterpieces then I really don't see an issue. There is no right score for any specific VN. The whole point of a voting system is to gather scores from people with differing opinions. To tell people that their opinion is wrong because you think the scores are too high (or low) is borderline defeating the purpose of the scoring system to begin with. If they are just trolling then obviously they should be removed but if they are truthful in their opinion then what right does anyone else have to say they are wrong?Last modified on 2012-06-04 at 22:09
#17 by gabezhul
2012-06-04 at 22:05
@13: Errr... just no. I'm not arguing bout the basic notion, but a solid, working an interesting VN is not 5-6, it's 7-8. A solid, working and AVERAGE VN is another thing though...

Also, the voting system will NEVER be truly objective. I would say Soketsu's idea would give a relatively good result with tying vote-weight to the number of read VNs, but even that won't guarantee that someone with more VNs behind his back would give any more objective scores than a well-meaning newbie.
#18 by saxman-faust
2012-06-05 at 03:50
Why would you expect someone to vote objectively, anyway? I vote based on my own experience; if I didn't enjoy it, I won't give it a high score.

This moronic "trolldar" notion is the reason I avoid voting low on popular games, e.g. I really, really didn't enjoy FSN. Maybe it was because I had just read Ever 17 (which I did enjoy), but I didn't find FSN at all engaging. I would probably rate it at a 5 (so-so) simply because, while it didn't make me want to put it down, it certainly didn't make me want to continue.

So, am I a troll because I deviate from the aggregate?
Should my vote and opinion be degraded or discarded for such an asinine reason?

Fuck that.Last modified on 2012-06-05 at 03:51
#19 by space-ranger
2012-06-05 at 08:41
Nobody said anything about making votes count less just because they are different from the average for that particular title. Voting 1 should count if you feel it is that bad. My point was to make votes count less if most/all votes were too high or too low. It should trigger if somebody decides to vote 1 or 10 for everything.

However saxman-faust do stand out when giving "troll scores". It's only 1,375, which is the lowest I have seen so far.

By the way this debate is pointless as the idea of weighted votes has been discarded.
#20 by horseband
2012-06-05 at 14:44
What's my score?

I realize the idea has been shot down but I still wanted to discuss your idea and beliefs behind voting. Even if the idea may not get implemented I still wanted to comment on some of your ideas and thoughts.Last modified on 2012-06-05 at 14:45
#21 by space-ranger
2012-06-05 at 15:02
@horseband: 1,868, which is more or less in the middle of the normal range.

Reply

You must be logged in to reply to this thread.