Don't Crawl VNDB!

Posted in

#51 by kiru
2016-12-04 at 10:13
I'm not sure if this helps, but I also somewhat encountered this at rare times. Usually when I do a major search via tags and open a LOT of stuff. (Not at the exact same time, but you know, basically I'm actively clicking for a while)

But as a general issue or something I've never encountered this. So maybe this has something to do with sessions and them being handled a bit weirdly. Just a very random idea. In my case it was too rare to really pinpoint anything, and it could've just been a short server issue.
#52 by dk382
2016-12-04 at 11:08
I enabled some performance logging on the server so see if it's on the response generating end, but things are looking fine so far. There's still many places where slowness can occur, however. :(

If the slow loading is easily reproducible, could I get a screenshot of the network monitor? Especially the 'timing' tab of a slow resource is interesting. Chrome has a similar feature, but I'm not too familiar with that one.

EDIT: To illustrate, a screenshot like this would be great. :)
The site is actually running really good for me right now, but next time I notice some slowness I'll do that.
#53 by yorhel
2016-12-04 at 11:31
This threat triggered me to draw some graphs. Here's a CDF of the response times in the last 2 and a bit hours, measured at both the server itself and the proxy in front of it. That's looking fairly good, overall.
#54 by silence
2016-12-04 at 11:58
Well, I'm still using old Opera 12, and that was the problem. There are no issues on Chrome >_<
Still, you somehow modified vndb, and now it's slow for old Opera.
#55 by yorhel
2016-12-04 at 12:10
Aaah. Pretty unlikely that any recent change to the site itself would cause that, more likely a change in the HTTPS configuration. But if you're using an old browser then quite likely I can't fix that without compromising on security.

And seriously, don't use old browsers, ever. No amount of virus scanners will save you.
#56 by silence
2016-12-05 at 11:37
It's fine. My shield is Kaspersky. No problems so far. I'll wait for some good browser without that stupid Chrome engine.
#57 by bunny1ov3r
2016-12-06 at 00:11
Oh wow just realized this site is on https. I am not an expert on tech stuff, but isn't https a lot more expensive? Wondering how you are doing financially with this site without ads or donation. Maybe we should open a donation option in the future.
#58 by dk382
2016-12-06 at 03:08
As far as I know, https isn't inherently more expensive than http. It's not something you have to register for or pay for. It's just a protocol you use for greater security. The only way it might be more expensive is that it takes more server processing power to encrypt all the data. From what Yorhel has posted though, the servers have an abundance of CPU capacity so I wouldn't worry about that. Apparently things are going just fine and the affiliate links are pulling their weight in keeping the site solvent.
#59 by eacil
2016-12-06 at 05:04
I think he is talking about certificates but I think vndb use Let’s Encrypt which is free?
#60 by yorhel
2016-12-06 at 06:05
Indeed, now that Let's Encrypt exists I use those free certificates. Before that, though, I bought 2 certificates for two years for about 15 $/cert/year. That's plenty cheap enough. And for a relatively small site like VNDB, the processing power required for HTTPS is utterly negligible.
#61 by bunny1ov3r
2016-12-07 at 00:47
ok good to know.
#62 by dk382
2016-12-07 at 02:01
Things are running a little bit slow for me. I said I'd post this stuff when it happened so here it is.

Overall graph: link

Note that the gap between the main page and the image being loaded was just due to me not clicking the image for a few seconds. The page itself was slightly sluggish but not overly so. Images are taking around 3 to 6 seconds to load, though.

Here's details for the page load: link
And the details for the image load: link

Other images are loading much more slowly, it seems a little random. Right now my internet usage is close to 0% and speedtest gives me 40mbps, which is my bandwidth cap, and other sites are running well. It's just the images here that are really sluggish, going at around 40-80kb/s.

That's not unbearably slow but if I'm trying to quickly look through a bunch of images at once it's pretty inconvenient. It's overall something I probably would never have complained about if the subject never came up here.
#63 by yorhel
2016-12-07 at 17:11
Thanks. Comparing your screenshots with my logs, I can see that the generation times for those hits were pretty fast, but the transfer times not so much. Problem probably lies somewhere in the network, but that'd be pretty hard to locate. Quite likely it's somewhere at the server end because I've seen some odd longer loading times as well at rare occassions. But I hadn't been able to reproduce those, so your screenshots helped increase my confidence that it's a network issue and it's not just me. :)

I'll run some ping measurements for a day and see if I can find increased latency at certain times.
#64 by thefreeloader
2016-12-28 at 06:38
Muhahaha! It seems my power, Ultimate Crawler, has brought chaos upon this site. I must seal it! Arghhhh!

Reply

You must be logged in to reply to this thread.