Some love for the API

Posted in

#51 by micah686
2015-08-22 at 19:21
< report >@onkelsam, what a very strange coincidence.
I was also working on making a visual novel manager in C# this summer, and it was also my first program in that language, and I was learning as I went. Maybe we could work together?

My source is here:
https://mega.nz/#!yUdHyBiA!uKDoRVxa-YlJ1jYQsn39ZFcApdSDPq4jehzlw8lZLp0

(please note that it is buggy and unfinished, and my code looks like crap)Last modified on 2015-08-22 at 19:22
#52 by onkelsam
2015-08-23 at 09:36
< report >@micah, sorry but I won't have time. I'll start an internship next month and I only wrote my application to get a feel for c# as I will need it for said internship.
#53 by kelpie
2015-09-07 at 19:46
< report >For some time now (1-2 weeks, I think), I've been unable to use the API. All commands get an empty response. I was assuming a temporary failure somewhere between me and the server, but the problem has stretched on too long.

I haven't changed anything at all on my end. One evening, right in the middle of a session using the API, it just stopped working.

For reference, I'm trying to connect to api.vndb.org (which resolves to 188.165.233.33 for me--and which I _can_ successfully ping). My login command is:

'login {"client": "Offline VN List", "clientver": "0.0.1", "protocol": 1}\x04'

I've tried with all firewalls/antivirus software off; I've tried it inside a VM, in case something had broken in my copy of python; I even ported my client from python 3 to python 2, out of a combination of superstition and despair. I'm out of ideas.

Is there anything that might have happened in the last couple of weeks that would stop the API working for me? I would assume there'd be a storm of complaints if it was a general problem, but I remain hopeful that the problem can be diagnosed.
#54 by yorhel
2015-09-07 at 19:57
< report >Yeah, I'm seeing log activity from you:

[Mon Sep 7 21:41:54 2015] 2160 <ip>:<port>: Connection denied, limit of 5 connections per IP reached

Do you happen to have a bunch of python scripts still running in the background that are hogging a connection? This is quite likely to happen in a web server setup where your server spawns a new process for each request and doesn't clean them up properly.
#55 by kelpie
2015-09-07 at 20:01
< report >No, the connections are initiated manually, and there's been a long gap and the client shut down (and a reboot, besides) since the previous attempt, so there shouldn't be any zombie connections hanging around. I'll double-check to see if I did anything that broke connection re-use, but the first connection ought to work, regardless, shouldn't it?

Edit: also, I've received no throttling messages, in case some should have been sent.

Edit 2: Perhaps I've found the proximal cause.

The last relevant modification I made was two years ago when I added a check that refreshed the connection when it was timing out. Since I'm not successfully getting a connection (no reply to login), it's trying to open a new connection on every command, which would explain me hitting the connection limit.

So, the question becomes: what happened that suddenly caused me not to be able to connect? I'm hoping the answer isn't 'inscrutable (but persistent) network error'.Last modified on 2015-09-07 at 20:10
#56 by yorhel
2015-09-07 at 20:08
< report >You won't get a throttling message in that case - the server closes the connection before even attempting to read a command.

It might be possible that the TCP connection has died long ago and the server hasn't gotten a notification about it. I'll try to enable TCP keepalive later and see if that can be used to detect dead connections automatically.

As a temporary solution (before I go to bed), I've restarted the server so you should be able to connect again.
#57 by kelpie
2015-09-07 at 20:14
< report >Success! Thank you very much for your help.

Given the distance between my attempts, the server must have been keeping those dead connections around for several days, or possibly more than a week--perhaps not optimal. I am cognizant of the irony that my previous complaint was that connections weren't kept alive _long enough_...
#58 by yorhel
2015-09-08 at 17:12
< report >Okay, I've enabled TCP keepalive. Dead connections should be detected and cleaned up after a few hours.
#59 by zolty
2015-09-17 at 04:45
< report >@yorhel
I was wondering if you could add a developer/producer field to "get vn" which could be auto-generated like the date field is (developer of first non-trial release).

At the moment I have to dig through each vn's releases to get developer which makes everything more complex and results in throttling when trying to get information about a userlist of 300+ items, especially because "get release" can only take one vnid at once and not an array of id's like "get vn", I got so confused with the nested if's that I moved on to another feature for the meantime.
#60 by yorhel
2015-09-17 at 10:08
< report >VN producer info is not currently cached like the first release date is, so performance-wise the improvements won't be too significant. Implementation-wise it's just moving the complexity from your code to mine, which I can't say I'm very fond of. :-P

Considering the limited use case (you might want the first release, others will certainly want the same kind of aggregation as seen in the info box on VN pages, others will want something else entirely...), I won't add it for now. I can consider relaxing the throttling rules, but seeing it's already possible to fetch the entire database in a few hours I'm not convinced it's really that strict.
#61 by zolty
2015-09-19 at 22:35
< report >Alright, it would be nice to have that info easy to get though.

About throttling, I have worked around it now and I plan to cache pretty much everything so in the end it won't be a big deal. (sucks while debugging though)

Like micah686 and onkelsam, I'm also making my first C# app and I used onkelsam's app (t3599.48) as a base, making this app to do some things that can't be done (or done well) with the vndb website such as a "watch list" for releases from favorite producers and a tag filtering system with toggle buttons. Coding is fun!
#62 by wraithseeker
2015-10-19 at 13:44
< report >Hey I am having some issues with the API. I'm not getting any response when I use the login command.

I am sending this string using Ruby
'login {"protocol":1,"client":"testruby","clientver":0.1}'
#63 by yorhel
2015-10-19 at 13:54
< report >See my reply in #37, you're probably hitting the same thing. The 0x04 byte can be generated in Ruby with the string:
"\04"
#64 by wraithseeker
2015-10-19 at 15:02
< report >I added the EOT at the end and still can't get a response for some reason.

Do you have any clues that could be causing the issue?Last modified on 2015-10-19 at 15:02
#65 by yorhel
2015-10-19 at 15:41
< report >The server is sending a response now. If you're not seeing it it's likely that you're using a line-based buffer or something. The server does not include a newline in the response, so if your code is waiting for a newline, it'll wait forever. Configure whatever abstraction you use to return data on the EOT byte and it should work.
#66 by micah686
2015-11-05 at 08:27
< report >@zolty
Would you be at all interested in working together on a C# program? Feel free to contact me if you would like to consolidate our efforts.Last modified on 2015-11-05 at 08:34
#67 by leiven
2015-12-04 at 04:48
< report >My PHP5.6 client implementation (formerly available under the Fuwanovel vendor name) is available again and in a stable version.

link

All commands exposed through the API are available now. Feedback encouraged and appreciated.
#68 by yorhel
2015-12-29 at 14:41
< report >I've (finally) added support for TLS. Check d11#1 -> "Connection info" if you want to use it.
#69 by leiven
2015-12-31 at 06:07
< report >Nice!

I added a new SecureSocket class to support the connection through TLS to my PHP client:

link

Will document and recommend using it soon(ish).
#70 by fredthebarber
2015-12-31 at 21:49
< report >Thank you, yorhel, for the TLS feature, and thank you, micah686, for giving me a heads-up about it on GitHub. I've added TLS and Login support to the little C# client I maintain: link

Whether you're using it as reference library to build a real C# app, or god forbid issuing queries to the VNDB API using the command-line EXE I put in the Release folder, hit me up (ideally through an issue on GitHub) if you have any questions, suggestions, etc.,
#71 by zolty
2016-01-24 at 02:47
< report >@micah686
I wish to work on my program by myself so I can learn more about programming and I also don't have time to work on it right now (last 6 months of CompSci degree).
#72 by micah686
2016-01-24 at 07:06
< report >Would you be willing to share your source code so far, so I could see what you have, and work on implementing it into my program on my own?
What you are working on sound like a good feature I would like to add to my program, so if were willing to share your source, I would greatly appreciate it.Last modified on 2016-01-24 at 07:17
#73 by yorhel
2016-01-29 at 12:22
< report >API update: it's now possible to fetch vn screenshots.
#74 by leiven
2016-02-07 at 13:57
< report >Thanks for the update, kinda missed it, but added the new flag and field to my PHP client:

link
#75 by eacil
2016-02-16 at 00:44
< report >@leiven, I am using your client: can you tell me if there is a way to overcome the 100 results limitation? I can't load all my wishlist in my own db because of that.
I see that the only filter available is uid but that doesn't help...