Downtime, downtime...

Posted in

#1 by yorhel
2014-03-11 at 17:27
The server suddenly got unresponsive at around 17:00 CET and VNDB has been down for approximately 1 hour and 30 minutes. A reboot helped for some reason, but I'm still investigating what went wrong.

Doesn't look like we lost any data.
#2 by pendelhaven
2014-03-11 at 18:49
"the server's power supply over-heated, and good thing they had a spare part to replace it. however it took them more than 60 minutes due to company policies (whatever they are...) whereas it could have only took them 10 minutes if without said company policies."

"on the other hand, its client, yorhel, might be thinking that it might have been something worse than just simply that. worried about why it took them that long (90 minutes), there might have been more than just a simple power failure for the server."

you're probably just worrying too much.
#3 by madeking
2014-03-11 at 18:57
1hr30 of downtime, biggest announcement since october last year.

At least its given you something new to post in the announcement space, its certainly been a while. got to dust it off now and again just to check its still working!
#4 by yorhel
2014-03-11 at 19:01
@pendel: Your hypothesis is bullshit. :P My current guess is that either some software ran out of control and started spawning lots of processes (akin to a fork bomb) or that the hdd or filesystem was swamped and failed to recover.

EDIT: In either case, the crash didn't leave any useful trace in my logs or monitoring tools, so I won't be able to figure out what the cause was. Let's just hope it was a one-time thing. Well, if it turns out that the server is completely screwed, that'd give me a good excuse to do a complete reinstall. Been wanting to do that for a while now anyway.Last modified on 2014-03-11 at 19:15
#5 by tokizo
2014-03-11 at 19:19
Its been a while, i'm glad now that i can access vndb again.

ps: just 5s ago after this post, under contruction banner show again and i press f5 it is gone.Last modified on 2014-03-11 at 19:21
#6 by space-ranger
2014-03-11 at 20:57
is there a backup of the database itself? It would suck if the server caught fire or something if there isn't any backup at all.

There was a thread about a backup torrent at some point. The solution would be to make a backup using the API. Considering that nobody appears to have done that yet I say that we should rethink that strategy.
#7 by yirba
2014-03-11 at 23:21
Creating a backup using the API would be extremely tedious. A much simpler solution would be for yorhel to simply export the database (probably excluding things like users' passwords, if not the entire user table) and make it available to download.

Wikimedia actually offer database dumps for all their projects, including the English Wikipedia, which can actually be pretty useful if you want to run your own SQL on it.
#8 by takata
2014-03-11 at 23:59
I didn't know you had physical access to the server yorhel. o.o

When I noticed the "under construction" banner, I thought you were upgrading something or adding a feature.
#9 by habluka
2014-03-12 at 03:19
Why would you ever expect vndb to ever get new features/upgrades?
#10 by loctar87
2014-03-12 at 06:12
Today is "Patch Tuesday", if your running a windows server. That's as likely a cause as anything.
#11 by yorhel
2014-03-12 at 06:20
I have automatic daily incremental backups to my fileserver at home. If the datacenter gets struck by a meteor we'll only lose one day worth of data at most. I'm definitely not going to provide raw database dumps for download (certainly not without a lot of censoring, making the dump hard/impractical to recover from), but I can provide encryped dumps if you think two servers isn't enough replication. I would still be the weakest link in that case.

I didn't know you had physical access to the server yorhel
I don't, but OVH allows remote reboots with their control panel (most dedicated hosting services can do this).

if your running a windows server
NEVAH!Last modified on 2014-03-12 at 06:30
#12 by abyssaleros
2014-03-12 at 09:38
Good to hear that apparingly no data was lost.

Although I have to confess that for a few moments I had the hope that some highly needed updates were coming, like a quick way to search in one owns Visual Novel List after On loaned novels (I never found a way), have statistics in filtering novels (and/or releases), like how many japanese Visual Novels are within the database if you filter out every shitty OELVN and so on ...

But hope never dies.^^
#13 by pendelhaven
2014-03-12 at 12:09
I guess we'll never know the cause. not that it really matters anymore.
#14 by space-ranger
2014-03-16 at 00:13
I would still be the weakest link in that case.
Good thing this isn't TV, because that would mean you would be history.

I'm definitely not going to provide raw database dumps for download (certainly not without a lot of censoring, making the dump hard/impractical to recover from)
A censored version shouldn't be that difficult to make. Just loop all users and replace everything with default settings, except possibly account name. This should make it possible to start up a server with everything preserved, except nobody can log in and everybody will have to make new accounts. The board history and all VNs will be preserved, which is the important part.

All votes and lists should also be removed. Maybe they should only be removed if the owner selected to hide them.

if your running a windows server
Windows and server doesn't belong in the same sentence. Besides windows is pretty close to being the only OS, which can't run the server software. It's written for linux, but I managed to get it to run on a mac. Windows on the other hand isn't something I would even attempt as it lacks everything the server needs.
#15 by justinizhere
2014-03-16 at 11:46
I would still be the weakest link in that case.
Good thing this isn't TV, because that would mean you would be history.
I see what you did there.
#16 by warfoki
2014-05-01 at 01:02
Bit of a necro, but this seemed to be the most appropriate thread for it. VNDB is awfully unresponsive right now. No 404 error as of yet, but pages are loading for minutes (!) and I got the following gem when I wanted to edit v8300:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

EDIT: Also, something's up with the search-function.

ギャルビッチなドS幼馴染みに調教されて奴隷になったボク gives me zero results, even though I've literally just copied it from "original title" field of v14702. Tried it several times, on two computers, before and after flushing the cookies, nothing. Weird.Last modified on 2014-05-01 at 01:59
#17 by yorhel
2014-05-13 at 06:46
Bit of a necro myself.

VNDB is awfully unresponsive right now.
Every first day of the month, close to midnight CET, a long-running cron job does some database checks and cleanup. The database is an order of magnitude larger now than when I wrote that script, so it has a tendency to slow things down and sometimes even crash parts of the site... I need to fix that sometime. >_>
#18 by pendelhaven
2014-09-01 at 14:31
^ it just did right now. but the downtime is (probably) relatively small so meh.
#19 by yorhel
2014-09-01 at 14:51
I changed the time that the crons would run, it's now at noon GMT. So that I'm actually awake when it runs. But yeah, again a lot of sluggishness and downtime, I need to fix that sooner rather than later...


You must be logged in to reply to this thread.