Another Server Move
|#1 by Yorhel|
2017-11-04 at 07:54
|< report >It's been 3 years since the last server move, so I was getting the itch again to get an upgrade. In particular, VNDB has been on old rotating drives for its entire lifetime, and I've started noticing a few latency spikes here and there. A move to SSDs would be welcome.|
Here's the new server specs.
CPU: Intel Xeon CPU D-1531 @ 2.20GHzYou can find the old server specs at t5808 for comparison. And indeed, 2x250GB is not a whole lot of space. I had to reorganize a few things in order to make that fit. I actually wanted a server with more beefy disks, preferably an SSD/HDD hybrid, but those options were either unavailable or more than twice as expensive. So this will have to do.
Disks: 2x250GB SSD
Unlike all previous server transfers, this announcement comes *after* we've moved. So, yes, welcome to the new VNDB. There was no need for a "read-only" mode this time, and there has been at most 1 minute downtime or so. Unless I made a silly mistake somewhere and failed to notice that something is horribly broken, I think I'm getting slightly better at these transfers. :-)
The API might have (had) some unavailability issues because the DNS may need some time to propagate. Please complain loudly if anything's broken.
UPDATE 2018-04-29: Moved again, see t9845.12.Last modified on 2018-04-29 at 09:13
|#2 by hinoe|
2017-11-04 at 07:57
|< report >But do all the pics even fit those SSDs? :P|
|#3 by elriel|
2017-11-04 at 08:28
|< report >As someone whose job description has included a lot of web development, I can say that the images don't really need to be present on the same server as the website logic.|
So whether or not they fit on this server is not really relevant. You can always stuff them into a CDN (or another server) to radically reduce the storage and bandwidth requirements on your main server.
Doing this can cause some minor extra complexities in managing the system, but nothing major. However, it's a pretty common choice for image-heavy sites.
Anyway, good job at the server move! It's not simple to get everything right on the first try.
|#4 by Yorhel|
2017-11-04 at 08:37
|< report >Heh, the entirety of VNDB (database + images + whatever) fits into RAM on that server. Fortunately, because that makes things a whole lot easier - I have experience with hosting annoyingly large sites too. The "reorganization" I mentioned had more to do with the other stuff that I hosted on the old server.|
|#5 by hinoe|
2017-11-04 at 08:41
|< report >elriel, that was actually a joke and a reference to a conversation I had with yorhel on IRC several months ago. Considering it's pretty much an in-joke, I'm not entirely surprised it kind of went over your head. :)|
|#6 by Ileca|
2017-11-04 at 08:42
|< report >He needs room to access his porn at any time everywhere on the planet. How understandable.|
The kind of content that goes out of control very fast and 500go is indeed not a whole lot of space.
|#7 by entra|
2017-11-04 at 12:43
|< report >VNDB is already one of the quickest sites I browse, so to see it potentially get faster is pleasant news! Thank you (and all the other contributors!) for all you work creating and maintaining VNDB, to many it's a very valuable site.|
If you don't mind, purely because I'm curious, could you briefly outline how you achieved the move with no read only state? I'm only slightly technical but I'm guessing it went something like: Postgres slave on the new site -> promote to master -> change old server to a reverse proxy to the new server -> update dns?
|#8 by Yorhel|
2017-11-04 at 13:23
|< report >Kind-of, except I didn't bother with a replication setup. It went like this:|
1. Do a full rsync of all files (including raw postgres data files) to the new server while the old one was still running.
2. Shut down all services on the old server, so that all data gets flushed cleanly and nothing new gets written anymore.
3. Run the rsync again to update the new server with a consistent state of the old server. This took only like 30 seconds.
4. Meanwhile, start a reverse proxy on the old server.
5. Start all services on the new server.
VNDB was down during step 2 till 4; but I had everything prepared so it didn't take much time. And rsync has saved my life yet again.
|#9 by thefreeloader|
2017-11-08 at 04:17
|< report >yorhel, you rock! I wish to donate my blood.|
|#10 by rockleevk|
2017-11-08 at 17:34
|< report >YAY! VNDB SSD Edition.|
Congrats on the upgrade.
|#11 by reo|
2017-11-09 at 22:24
|< report >No wonder recently I found that this site loaded faster than ever. Nice upgrade and thanks.|
|#12 by Yorhel|
2018-04-29 at 09:13
|< report >We just had another server transfer, because 2x250G was not enough for my other stuff, after all. We're now running on exactly the same hardware, just with bigger SSDs:|
CPU: Intel Xeon CPU D-1531 @ 2.20GHz
Disks: 2x500GB SSD
I'm starting to get used to moving VNDB around.
|#13 by kiru|
2018-04-29 at 10:40
|< report >I liked the moving page. My rage was quelled by it.|
|#14 by pendelhaven|
2018-04-29 at 11:17
|< report >to think this all started on a wasteful pentium D...|
|#15 by Ileca|
2020-05-21 at 00:05
|< report >How much bandwidth vndb consumed averagely every month for the past year?|
I remember you made public your server monitoring tools but I can't remember where is the damn link.
|#16 by Yorhel|
2020-05-21 at 04:51
|< report >About 2.5 TiB/month on average. link|
|#17 by Ileca|
2020-05-21 at 06:12
|< report >That's huge. I might have underestimated the display of screenshots.|
I guess I should have asked for the number of visitors per day to put things into perspective though it will not make any difference as I don't know what's considered "average".
Weird, I thought the flagging feature would draw some peaks but I see nothing. Now, I read on a previous thread from 2012 that you have another server for images so maybe those screenshots are an incomplete picture.
|#18 by Yorhel|
2020-05-21 at 06:21
|< report >I don't have a separate server for images anymore, it's all on a single server now. And it's not that bad, really. VNDB is pretty light in terms of bandwidth, even with the screenshots. Even if you'd asked about visitors/day, I haven't a clue because I don't track that. :) All I know is that we get about 6-8 pageviews/second on average (I should graph that), and with 1 MiB/s average bandwidth that comes down to ~150 KiB/pageview.|
The vast majority of pageviews comes from visitors who aren't even logged in, so I never expected the image flagging feature to have much of an effect.
You must be logged in to reply to this thread.