VNDB 2.25: High Speed, Low Velocity

Posted in

#26 by space-ranger
2015-10-26 at 13:52
I just thought that he said "bottleneck in sql" so I thought an increase in computing power might eliminate the bottleneck.
That's not wrong and if the hardware is too slow to cope with the load, then a hardware upgrade like that would be the way to go, at least compared to the multiple server setup, which was in the question. However with the 7% server load, the "best upgrade" discussion is a theoretical one and not a practical one as no upgrade will be needed for quite a while.

To add to the confusion, somebody (google?) is working on a networked database server system to allow spreading database load on multiple servers while keeping the servers in sync. Assuming the server load increases with the current rate, by the time VNDB needs new hardware in the far future, that might have developed into a mature setup, which would also be an option to consider. With changes like that and possibly yet to be published options, this entire discussion on how to upgrade VNDB hardware is kind of pointless. Interesting to some degree, but not useful at all for VNDB.
#27 by yorhel
2015-10-26 at 16:44
Distributed databases and the research into that area has existed for decades, and they're in a somewhat usable state. I'm not expecting any major breakthroughs there.

That said, the entire premise is false; VNDB in its current state should pretty easily scale to multiple servers. Around 99% of all pageviews only need read-only database access, and PostgreSQL has mature master-slave replication built right into it. I'm not forseeing any serious scalability issues for VNDB in the next several decades if we keep growing like we have. :-)
#28 by lmportant
2015-10-26 at 16:50
You never know if there will be a sudden spike...
#29 by space-ranger
2015-10-26 at 20:10
PostgreSQL has mature master-slave replication built right into it
*facepalm*
You are right (you usually are regarding databases). I didn't think of keeping one master, which will be the only one accepting edits and then read only clones. Yeah, that would scale quite nicely to a level far beyond anything I can imagine VNDB will become within decades, or possibly ever.

Actually with that amount of potential for server capacity, shouldn't we make more use of the CPU power... because we can :P

You never know if there will be a sudden spike...
True, but how much are you willing to pay to avoid an extra second delay in such rare cases? The same question remain if you mean a sudden increase in baseload without it falling again. I think we will just have to live with the risk that a pageload can be delayed for what seems like no reason. It might even be a delay between you and the server rather than server load. It's not like it's realistic to do anything about the latter either.Last modified on 2015-10-26 at 20:12
#30 by gabezhul
2015-10-26 at 20:41
Also remember that most of the page-loading delays you see on 99% of pages are not actually from the actual server, but because of all the other extraneous stuff that is also on the pages. Case in point, the main page of your average gaming website usually has to simultaneously load from:
-The server that houses the site itself
-Facebook, Twitter and whatever other social media sites are shown (usually as part of a widget or a comment system)
-Ads from Google
-Ads from everywhere else
-Other external sources, such as avatar pictures from image sites or videos from YouTube
-The backdoor script that lets that nice NSA agent peep at you through your webcam RIGHT NOW

Compared to that, VNDB is not only really lightweight when it comes to design, it also lacks all of these other hangers-on that each have their own individual latency and server-load that can slow down the page. So yeah, unless Yorhel suddenly decides to turn the site policy around 180° and implements a bunch of ads and widgets overnight, we are cool.
#31 by skorpiondeath
2015-10-30 at 17:54
@yorhel: thanks for the improvement my pages now loads super fast!!
If we have a spike just ban traumatizer DB load will suddenly lower by 47,5%.
#32 by space-ranger
2015-10-30 at 23:35
I don't think banning any one person would really prevent spikes. Sure u78535 edits a lot, but no more than what the DB should be able to handle. Besides if I had to kick out users with the message "come back in a few seconds", I would rather do that to people using the database in read only mode and not prevent any edits.
#33 by skorpiondeath
2015-10-30 at 23:46
I don't think banning any one person would really prevent spikes

Lol I thought that it was obvious that I was joking!! :D
#34 by space-ranger
2015-10-30 at 23:54
I thought you might be, but with more than 10k edits, you never know how people think. I figure that if I reply decently on something like that, people either get happy with a decent reply or they start laughing due to the joke. It's a win-win ;)

Actually the only way to screw up in such a case is to pretend it's a joke when people re genuine. That would be like laughing at a picture of somebody because they showed an ugly pranked picture and then it turns out that they really do look like that.Last modified on 2015-10-30 at 23:56
#35 by traumatizer
2015-10-31 at 00:09
Wait I've actually made that many edits just in a year? Hmm, maybe I should just retire from VNDB, and spend the rest of my life peacefully spectating the new, young blood & the others editing like crazy. Nah, I think I'll continue doing my job, but more slowly. I think I've done edits even without realizing, like it's become a routine for me... Or it's as if doing edits has always been a part of my life.
But holy hell, I've pretty much made a lifetime amount of edits!! :D A-Am I supposed to be happy about this achievment?Last modified on 2015-10-31 at 00:10
#36 by skorpiondeath
2015-10-31 at 00:12
Actually the only way to screw up in such a case is to pretend it's a joke when people re genuine.
Yes space-ganger I got it probably I should have specified square 50% in performance improvement since 47,5% made it look so serious as I did some actual case study!! My bad :D As I side note I appreciate traumatizer hard work, just in case anyone else got it wrong! :P

Yes be happy traumatizer :D Retirement is not an option for you!!Last modified on 2015-10-31 at 00:14
#37 by space-ranger
2015-10-31 at 00:57
Hmm, maybe I should just retire from VNDB
Do you honestly think you can convince us that you will do that? VNDB itself has become a game for you and it's an awesome one because you will never end up throwing it away because "I have done it all". Just keep on "playing" and enjoy yourself.

Yes space-ganger I got it probably I should have specified square 50% in performance improvement since 47,5% made it look so serious as I did some actual case study!! My bad :D
I should have spotted that. It's not 47,5%. If I start calculating on the numbers based on the server CPU load, the reduction is 72%. Assuming people make the same amount of searches, the server is 257% faster. 47,5% is way off and way too specialistic. Darn, I should have noticed this right away and written an even better reply :(

I question if that is the real numbers for improvements though. They aren't measured as accurately as I usually do when figuring out the effect of code optimization. Still even if we are careful and assume ±20% error, those numbers are still really impressive. The assumed "same page view load" is stretching the comparison too because we don't know if that is the case.

Also you have an issue with names. Considering you mistype even when it's on the screen and you can copy paste if needed, that's an even bigger fail than me not noticing your bogus number :PLast modified on 2015-10-31 at 00:59
#38 by skorpiondeath
2015-10-31 at 14:04
Also you have an issue with names. Considering you mistype even when it's on the screen and you can copy paste if needed, that's an even bigger fail than me not noticing your bogus number :P
Indeed space-ranger, indeed! (ctrl-c is my friend)

Reply

You must be logged in to reply to this thread.