Some love for the API

Posted in

#26 by yorhel
2014-08-19 at 05:47
< report >Sorry, API crashed. It's working again now. is the old beta server, and has been down for ages. That one won't work anymore. :)
#27 by yorhel
2014-08-21 at 07:32
< report >Some recent updates:
- The get vnlist/votelist/wishlist commands can now be used to get the lists for anyone who has their list public.
- Maximum number of results for get vnlist/votelist/wishlist has been increased to 100.
- d14 now has daily vote dumps.
#28 by ravosu
2014-08-21 at 14:19
< report >@27
Awesome! You say 24 hours, but is it dumped at midnight? I would like to setup a cron task to refresh my DB a while after you create the dump.
#29 by yorhel
2014-08-21 at 14:48
< report >Midnight in GMT, yeah.
#30 by ravosu
2014-08-21 at 17:55
< report >Thank you. I don't see this in the documentation but maybe it's there:
Is there any API call to get username by uid and/or uid by username?
#31 by kuroonehalf
2014-09-15 at 18:08
< report >Hey guys. I'm trying to build a userscript to work on vndb with the purpose of previewing the cover images of novels when mouse hovering over their respective links.

However, I have absolutely zero previous experience with APIs and I'm having trouble interpreting the API documentation. Could someone please ELI5 how I'd use the "get vn image" command in either js or jquery, preferably with some code examples? I've been scouring all sorts of pages but I still haven't figured this one out. Any help would be greatly appreciated.Last modified on 2014-09-15 at 18:11
#32 by yorhel
2014-09-17 at 06:25
< report >@kuroonehalf: The API is not directly accessible from Javascript in the browser. You'll either have to write a browser extension or some server-side code to interface with the API.
#33 by kuroonehalf
2014-09-17 at 21:58
< report >Hm, that's a few notches above what I can do. I believe I read somewhere you frown upon crawling. Would you be against me crawling (or is this perhaps scraping?) the html of the pages to obtain the cover image links? That seems like the most straightforward way to implement this thing, and in either case it should be less intensive on the server than manually opening up and loading all of the VN results in a search, which is what I mostly do currently (since I memorize most novels by cover rather than by title).
I'm totally fine if it's only for personal use and not being able to share the code, but I'd really like a better way to browse stuff.

ps: A friend had suggested I ask you about a query based image response system, wherein typing something like "link" would return "link". What's your stance on that?Last modified on 2014-09-18 at 02:15
#34 by yorhel
2014-09-19 at 06:13
< report >I see there's a script already, at t5325.11. Which is a fine solution.

ps: A friend had suggested I ask you about a query based image response system
I'm not interested in providing an HTTP API at this point.
#35 by kuroonehalf
2014-09-19 at 21:50
< report >Oh wow, I had no idea someone had made one. Welp.

I finally got mine fully functioning this afternoon and it does about the same thing, except it has improved preview functionality and I'm looking into adding caching into the mix (so that I don't constantly have to request the same HTML over and over for pages I've been to).

I might just post it there too once I take care of these loose ends.Last modified on 2014-09-20 at 00:45
#36 by sinny
2014-12-04 at 06:50
< report >Hello. I wanted to work with the API using PHP to try programming with TCP/IP. I opened a connection to the server and sent a login request, but get an error (104: Connection reset by peer) when reading from the socket. I've tried with and without socket lingers and keep-alives and encoding messages in utf-8. Would anyone be able to help a beginner determine what's causing the error? You can view my code here if you're interested: link
#37 by yorhel
2014-12-04 at 07:19
< report >@shinny: There are two things that go wrong on your end. The first, and why you didn't get a normal response the first 5 times, is that you should use double quotes in PHP for the \x04 byte to actually get interpreted. That is, '\x04' = the string as you see it, and "\x04" = the byte as we want it.

The second problem, and why you're seeing the disconnects, is that, if your script does not get a response at all, it will keep waiting indefinitely, and it doesn't seem like long running scripts are being killed on your machine. So after a few runs of your script, you'll have more than 5 connections open to the API, at which point it will immediately disconnect you if you try more.

And the keepalive / session linger settings shouldn't matter at all. I've no idea how PHP handles character data, but I suspect your strings are already UTF-8 if there's some non-ASCII in it, so the encode_utf8() probably isn't necessary.Last modified on 2014-12-04 at 07:22
#38 by yorhel
2014-12-04 at 08:31
< report >I was sure PHP had some better abstraction to do TCP, so I looked around a bit. fsockopen() seems easier to use, and allows you to use stream_get_line(), which, if it does what it documents, seems perfect for reading any message up until the \x04 byte (i.e., pass "\x04" as $ending, and a large enough $length).
#39 by sinny
2014-12-07 at 17:46
< report >@yorhel: Thanks for your help. I managed to get it working and see the 'ok' string. I'll experiment with those functions you mentioned when I get the chance since they do look to simplify the program. Appreciate all the work you've done!
#40 by jfaassen
2015-01-03 at 12:41
< report >@yorhel @sinny here's a PHP implementation of a VNDB client.


It does the basics (login, dbstat, get novel/release/producer/character data). Other methods (votes etc) are not yet available, but they can be implemented easily using raw requests. And PRs are welcome! :-)

#41 by fredthebarber
2015-04-11 at 16:49
< report >Someone was asking about your API over on another forum, so I slammed together a quick POC C# app to do the basics (connect, login, and then take arbitrary queries on the command line and dump the results): link

I promptly got a question about why I wasn't supporting username/password login, but I don't want to do that without either first establishing a server authenticated SSL connection or without you providing some reasonable authentication handshake that won't expose the password in the clear. The easier thing is probably the former, so, any chance of you adding an SSL-protected endpoint for the API? You can use the same cert you're using for the website (the hostname won't match by default, but I can just pass "" as the hostname for validation despite connecting to "", so it'll match your cert...)
#42 by yorhel
2015-04-12 at 06:16
< report >Hmm yes, I used to have the excuse that passwords were already sent over the wire in plain text with the web interface, but now that we have HTTPS that doesn't apply anymore. Adding SSL to the API is quite a pain in the current server framework. I did start on a rewrite of the API server some time ago, but lost interest half way. I'll see if I can start up that project again sometime.
#43 by jinhamazura
2015-04-28 at 22:45
< report >@11 Do you know if I should execute any command before this for use python API?
from vndb import VNDB
#44 by yirba
2015-04-29 at 08:36
< report >The Python library appears to import its own dependencies, so I wouldn't expect any prerequisites to be necessary.

Have you tried using the library yourself? If not, doing so is an easy way to find out if it works. Otherwise, if you have tried it and are experiencing issues, it would be helpful to know specifically what these issues are and/or what error messages you're getting.
#45 by jinhamazura
2015-04-29 at 13:18
< report >File "", line 6, in <module>
from vndb import VNDB
ImportError: cannot import name VNDB
seems I need to have something called vndb?
#46 by jinhamazura
2015-05-03 at 14:57
< report >I'm working without using it, but now I have another question, is there any way to get VN data using the name instead of the id?
#47 by leiven
2015-06-24 at 11:00
< report >And another PHP implementation, which already covers most commands apart from the "set" type (simply didn't bother yet) and some of the more complex flags.

Includes caching, and can be used as simple or advanced as one would want.


See BaseClient + Command namespace for the advanced way, and the ClientFacade class for the simple way, to find out how to send commands to the API.

Extensive readme to follow. As usual, PRs welcome.Last modified on 2015-06-24 at 11:00
#48 by onkelsam
2015-08-15 at 20:47
< report >Hello, I created a little program (in c#) to keep track of your VNs. It let's you fetch the information from VNDB (by ID) and stores it locally in a .json file. Then you can start and categorize it without searching through your harddrive.

It also downloads the VN's screenshots and character images and let's you cycle through them.

@yorhel is that considered crawling VNDB?

Here two screenshots of it: Imgur (yes, it's ugly...).

Source It's my first program in c# and I lack experience, so don't expect a lot.

@fredthebarber thank you for your c# app it helped me a lot!Last modified on 2015-08-15 at 23:20
#49 by yorhel
2015-08-16 at 06:46
< report >@onkelsam: If you're fetching web pages from VNDB, then yes that's crawling. For the screenshots it's understandable, because those are not accessible through the API at the moment. But why did you crawl for the character images? You can get those links from the API!
#50 by onkelsam
2015-08-16 at 11:36
< report >Sorry yorhel, didn't think about it. Changed it so links to character images are received from the API.