| #1 by Yorhel 2017-11-04 at 07:54 | < report >It's been 3 years since the last server move, so I was getting the itch again to get an upgrade. In particular, VNDB has been on old rotating drives for its entire lifetime, and I've started noticing a few latency spikes here and there. A move to SSDs would be welcome. Here's the new server specs. CPU: Intel Xeon CPU D-1531 @ 2.20GHz You can find the old server specs at t5808 for comparison. And indeed, 2x250GB is not a whole lot of space. I had to reorganize a few things in order to make that fit. I actually wanted a server with more beefy disks, preferably an SSD/HDD hybrid, but those options were either unavailable or more than twice as expensive. So this will have to do.RAM: 32GB Disks: 2x250GB SSD Unlike all previous server transfers, this announcement comes *after* we've moved. So, yes, welcome to the new VNDB. There was no need for a "read-only" mode this time, and there has been at most 1 minute downtime or so. Unless I made a silly mistake somewhere and failed to notice that something is horribly broken, I think I'm getting slightly better at these transfers. :-) The API might have (had) some unavailability issues because the DNS may need some time to propagate. Please complain loudly if anything's broken. UPDATE 2018-04-29: Moved again, see t9845.12.Last modified on 2018-04-29 at 09:13 |
| #2 by hinoe 2017-11-04 at 07:57 | < report >But do all the pics even fit those SSDs? :P |
| #3 by elriel 2017-11-04 at 08:28 | < report >As someone whose job description has included a lot of web development, I can say that the images don't really need to be present on the same server as the website logic. So whether or not they fit on this server is not really relevant. You can always stuff them into a CDN (or another server) to radically reduce the storage and bandwidth requirements on your main server. Doing this can cause some minor extra complexities in managing the system, but nothing major. However, it's a pretty common choice for image-heavy sites. Anyway, good job at the server move! It's not simple to get everything right on the first try. |
| #4 by Yorhel 2017-11-04 at 08:37 | < report >Heh, the entirety of VNDB (database + images + whatever) fits into RAM on that server. Fortunately, because that makes things a whole lot easier - I have experience with hosting annoyingly large sites too. The "reorganization" I mentioned had more to do with the other stuff that I hosted on the old server.Last modified on 1970-01-01 at 00:00 |
| #5 by hinoe 2017-11-04 at 08:41 | < report >elriel, that was actually a joke and a reference to a conversation I had with yorhel on IRC several months ago. Considering it's pretty much an in-joke, I'm not entirely surprised it kind of went over your head. :) |
| #6 by Ileca 2017-11-04 at 08:42 | < report >He needs room to access his porn at any time everywhere on the planet. How understandable. The kind of content that goes out of control very fast and 500go is indeed not a whole lot of space. |
| #7 by entra 2017-11-04 at 12:43 | < report >VNDB is already one of the quickest sites I browse, so to see it potentially get faster is pleasant news! Thank you (and all the other contributors!) for all you work creating and maintaining VNDB, to many it's a very valuable site. If you don't mind, purely because I'm curious, could you briefly outline how you achieved the move with no read only state? I'm only slightly technical but I'm guessing it went something like: Postgres slave on the new site -> promote to master -> change old server to a reverse proxy to the new server -> update dns? |
| #8 by Yorhel 2017-11-04 at 13:23 | < report >Kind-of, except I didn't bother with a replication setup. It went like this: 1. Do a full rsync of all files (including raw postgres data files) to the new server while the old one was still running. 2. Shut down all services on the old server, so that all data gets flushed cleanly and nothing new gets written anymore. 3. Run the rsync again to update the new server with a consistent state of the old server. This took only like 30 seconds. 4. Meanwhile, start a reverse proxy on the old server. 5. Start all services on the new server. VNDB was down during step 2 till 4; but I had everything prepared so it didn't take much time. And rsync has saved my life yet again. |
| #9 by thefreeloader 2017-11-08 at 04:17 | < report >yorhel, you rock! I wish to donate my blood. |
| #10 by rockleevk 2017-11-08 at 17:34 | < report >YAY! VNDB SSD Edition. Congrats on the upgrade. -rockleevk |
| #11 by reo 2017-11-09 at 22:24 | < report >No wonder recently I found that this site loaded faster than ever. Nice upgrade and thanks. |
| #12 by Yorhel 2018-04-29 at 09:13 | < report >We just had another server transfer, because 2x250G was not enough for my other stuff, after all. We're now running on exactly the same hardware, just with bigger SSDs: CPU: Intel Xeon CPU D-1531 @ 2.20GHz RAM: 32GB Disks: 2x500GB SSD I'm starting to get used to moving VNDB around.Last modified on 1970-01-01 at 00:00 |
| #13 by kiru 2018-04-29 at 10:40 | < report >I liked the moving page. My rage was quelled by it. |
| #14 by pendelhaven 2018-04-29 at 11:17 | < report >to think this all started on a wasteful pentium D... |
| #15 by Ileca 2020-05-21 at 00:05 | < report >How much bandwidth vndb consumed averagely every month for the past year? I remember you made public your server monitoring tools but I can't remember where is the damn link. /curious |
| #16 by Yorhel 2020-05-21 at 04:51 | < report >About 2.5 TiB/month on average. link |
| #17 by Ileca 2020-05-21 at 06:12 | < report >That's huge. I might have underestimated the display of screenshots. I guess I should have asked for the number of visitors per day to put things into perspective though it will not make any difference as I don't know what's considered "average". Weird, I thought the flagging feature would draw some peaks but I see nothing. Now, I read on a previous thread from 2012 that you have another server for images so maybe those screenshots are an incomplete picture. |
| #18 by Yorhel 2020-05-21 at 06:21 | < report >I don't have a separate server for images anymore, it's all on a single server now. And it's not that bad, really. VNDB is pretty light in terms of bandwidth, even with the screenshots. Even if you'd asked about visitors/day, I haven't a clue because I don't track that. :) All I know is that we get about 6-8 pageviews/second on average (I should graph that), and with 1 MiB/s average bandwidth that comes down to ~150 KiB/pageview. The vast majority of pageviews comes from visitors who aren't even logged in, so I never expected the image flagging feature to have much of an effect. |
| #19 by Yorhel 2023-02-16 at 09:23 | < report >We just survived (I hope) another move. I saw an offer for a server with similar specs for half the price and couldn't resist. CPU: Intel Xeon CPU E5-1630 v4 @ 3.70GHz RAM: 64GB Disks: 2x480GB Double the RAM, but it's not like VNDB uses that. I moved query.vndb.org to my little file server at home; that one has a more powerful CPU so heavy queries should run a little faster now. |
| #20 by -pan- 2023-02-16 at 16:01 | < report >I see the pictures uses .jpg format. A suggestion (if the need arises) would be moving to .webp. Nowadays most sites on the internet uses .webp to use half the size of .jpgs are basically the same quality (or sometimes even better quality). So you may consider using that in the future, if you wish to use less space and band for pictures. |
Reply
You must be logged in to reply to this thread.