VNDB 2.25: High Speed, Low Velocity
|#1 by yorhel|
2015-10-25 at 09:14
|< report >Somewhere in the next few minutes to hours, I will be pushing out a major update to the internal database structure and code. I expect that VNDB will be down for 15 minutes or so.|
This is an internal rewrite that I have been wanting to do for several years now. It should provide a good performance improvement to the page loading times and allow us to scale the database to many more entries without overloading the server.
Since this update touches a lot of core code, I expect that, despite my attempts to test everything, some functionality might be broken after the update. In the worst scenario, I might have to do a rollback of the complete database to the point before I pushed the update, but I hope that won't be necessary.
UPDATE: And it's live. Let the mayhem begin!
If you notice anything that seems to be different than before, please let me know asap so I can fix it. Except if that difference is "the site seems faster", of course.
|#2 by gapho|
2015-10-25 at 10:03
|< report >oh noLast modified on 2015-10-25 at 10:03|
|#3 by gabezhul|
2015-10-25 at 10:11
|< report >The site just reached through the screen and tried to eat my cat. Is that normal?|
|#4 by yorhel|
2015-10-25 at 10:13
|< report >Tried? You mean it failed? Damn, no, that's definitely a bug.|
|#5 by gabezhul|
2015-10-25 at 10:14
|< report >Okay, just checking. :P|
Jokes aside, nice to see the site still developing even after all these years. Nice job, have a virtual cookie. :)
|#6 by chrupky|
2015-10-25 at 10:48
|< report >Does better code base mean we can hope for new features? :>|
|#7 by ff80c38|
2015-10-25 at 10:58
|< report >The vote stat histograms look a bit different now, probably because rounding doesn't seem to be consistent anymore. 9.5 is counted as 10, whereas 8.5 and 7.5 are counted as 8. That pattern also continues for the other .5 numbers.|
|#8 by yorhel|
2015-10-25 at 11:27
|< report >@chrupky: I do intent to work on VNDB more actively for a while, but that doesn't necessarily mean new features. The way I prioritize my work has a tendency to be the opposite of what most people are interested in. >.>|
@ff80c38: Nice catch, fixed.
|#9 by takata|
2015-10-25 at 11:35
|< report >Performance improvements? vndb was already the fastest or one of the fastest-loading sites I regularly visit. Thanks anyway though. ^^Last modified on 2015-10-25 at 11:36|
|#10 by pendelhaven|
2015-10-25 at 11:55
|< report >^ totally agree with this. facebook doesn't even compare. yorhel, you're not... spending --more-- with this improvement right?Last modified on 2015-10-25 at 11:56|
|#11 by lbs21|
2015-10-25 at 12:11
|< report >Dang. VNDB was fast, now it's faster. Thanks for all the hard work you put into making this website great!|
|#12 by gabezhul|
2015-10-25 at 14:06
|< report >Hm. Yorhel, I don't think user discussions supposed to show up on the main page...|
|#13 by yorhel|
2015-10-25 at 14:19
|< report >Oops.|
|#14 by space-ranger|
2015-10-25 at 16:53
|< report >|
The site just reached through the screen and tried to eat my cat. Is that normal?With the high amount of 18+ titles in the database, I would say attempts at pussy eating would be quite normal.
I take that this update is the long talked about split of history and current version of vns/releasec/etc, meaning to look at a vn page, it will not have to search through old versions of vn pages to find the correct one. Less data to search through means faster search times. It's the only thing I can remember you have been talking about for years regarding performance.
Performance improvements? vndb was already the fastest or one of the fastest-loading sites I regularly visit.More performance is always a good thing. With an ever expanding database of new VNs and (hopefully) more traffic due to more VN readers, improved code is needed to even maintain the current load times on the same server hardware.
|#15 by yorhel|
2015-10-25 at 17:39
|< report >Prelimenary measurement results: This update has caused a drop in CPU usage from 25% to 7%. That's pretty sweet.|
I take that this update is the long talked about split of history and current version of vns/releasec/etcHeh, you have good memory. Yes, this is that update. But while I was changing the database schema I also did a full check on all the database queries on the popular pages, and improved the slow queries and added extra indices where that made sense (this is something that should be done every once in a while anyway). The majority of the performance improvement in this update comes from those optimizations rather than the database schema change, though that certainly helped as well.
improved code is needed to even maintain the current load times on the same server hardware.Case in point.
|#16 by vortexoblivion|
2015-10-25 at 22:54
|< report >Faster, the better as I can say :b|
|#17 by lmportant|
2015-10-26 at 01:49
|< report >"Server" This site is running on one machine?... Or multiple...Last modified on 2015-10-26 at 01:50|
|#18 by space-ranger|
2015-10-26 at 03:18
|< report >|
"Server" This site is running on one machine?... Or multiple...t5808
It's a single server as the bottleneck is the sql database and splitting that one into multiple servers would be quite tricky.
|#19 by lmportant|
2015-10-26 at 06:29
|< report >Maybe you can invest around 5 thousand and slap in a Nvidia Tesla to increase computing power...Last modified on 2015-10-26 at 06:39|
|#20 by pendelhaven|
2015-10-26 at 10:26
|< report >|
seriously, no. not when the CPU is just 7% busy.
|#21 by space-ranger|
2015-10-26 at 10:55
|< report >Something is wrong here. I agree with pendelhaven o_O|
I don't see any reason for investing in new hardware if the current one can handle the job just fine. Besides if I understand it correctly, VNDB runs on rented hardware, which makes custom hardware upgrades a bit tricky. The GPU solution looks like a much better solution than multiple servers though as a quick google search indicates an existing GPU version of PostgreSQL. Still doesn't make sense to spend money/time on an upgrade until the server load becomes a real issue.
Now that I think about it, I'm not even sure database lookup would be faster with a GPU. A GPU is not lightning fast as such, but exploits a huge array of GPU cores. This mean the end result could be the same search time, but adding the ability for multiple searches in parallel. If that's the case, then it totally makes no sense to upgrade until queues for the database becomes common.
|#22 by takata|
2015-10-26 at 11:16
|< report >Don't those GPGPU cards cost at least 1 or 2 times the cost of all of the server's hardware?|
|#23 by pendelhaven|
2015-10-26 at 11:40
|< report >It almost does. Which is why its not a good idea in the first place.|
|#24 by space-ranger|
2015-10-26 at 13:18
|< report >I would dare to say that GPGPU is brilliant, but only in specific cases. If you have a server load, which requires two servers, but you can't split a database between the servers for technical reasons, then companies goes for one super server even if it ends up being more pricy than two normal servers. However as long as one "regular" server is enough, any talk about using multiple servers or boost performance of that one server seems kind of silly. It would be like investing in a new fast car and still be stuck in traffic (read: more money for the same travel time).|
|#25 by lmportant|
2015-10-26 at 13:23
|< report >I just thought that he said "bottleneck in sql" so I thought an increase in computing power might eliminate the bottleneck... Shows how little I know..|