A New API

Posted in

#1 by Yorhel
2022-10-14 at 12:45
< report >It's been almost 13 years since the TCP API has been introduced and, despite its quirks and being rather unusual, even for its time, it has served us well. Still serving us, in fact. But time has moved on and raw TCP has become slightly more problematic over time on the modern internet, as every ISP and every piece of software assumes that HTTPS is the only protocol worth caring about.

Long story short, I've started working on a HTTPS-based API and you can find it at https://api.vndb.org/kana.

It's still in development, so expect missing functionality, bugs and potentially a few breaking changes here and there as I finalize the API. Aside from building on top of HTTPS, I also took this opportunity to fix some warts of the old API: the new filter system ties into the advanced search, field selection is now more granular and it has better support for fetching information from nested entries. This API should also be usable from browser extensions and user scripts, using it instead of scraping pages should help with avoiding the throttle limit.

My main focus so far has been on read-only data queries, but list management is also on the roadmap. I prioritize implementing features that people actually need, so by all means let me know what you're working on. Happy devving!
#2 by blahblah35
2022-10-14 at 19:09
< report >It's a nice news ! It's true that the TCP API wasn't the most straightforward thing to use, and that HTTPS-based API are mo easily implemented. I'll try to make something in python if I can find the time to do it.
#3 by visualnovelist-
2022-10-14 at 20:03
< report >Amazing news! I was planning to use the database as part of my thesis paper. It would be really helpful to have an easier source of data querying!

I just found one of your posts from 2019 about you wanting a dashboard tool to supplement the site with meaningful dashboards on the data. Was there any progress on that project that you are aware of?
#4 by impurepug
2022-10-14 at 20:36
< report >Was hoping to make an AMQ-esque game for visual novels, so an endpoint to return VNs that a user has read (to scrape for games to look for songs from) would be much appreciated!
#5 by Yorhel
2022-10-15 at 05:57
< report >
a dashboard tool to supplement the site with meaningful dashboards on the data
I think that refers to the query interface, which is probably also more suited for analytical queries than the API.
#6 by Yorhel
2022-10-15 at 08:36
< report >
so an endpoint to return VNs that a user has read
"POST /ulist" API added. I may have gone a little overboard, it's a lot more powerful than what you asked for. But it can only fetch public lists at the moment, authentication support will come later.
#7 by jinokigmi2102
2022-10-15 at 15:49
< report >sounds interesting. But personally, I will wait for the beta version, because I don't know anything about it
#8 by YuzuReader
2022-10-16 at 20:55
< report >Nice! So it will be possible to make an actual rng roulette for user list, finally! Thanks.
#9 by an0nym0us
2022-10-17 at 18:41
< report >It's much easier API to use than the TCP one. Thanks.

Is it possible to allow access to it in the Content Security Policy of the site so userscripts could use regular request APIs instead of GM_xmlhttpRequest?
#10 by Yorhel
2022-10-17 at 19:09
< report >
Is it possible to allow access to it in the Content Security Policy of the site
Good one, done.
#11 by ahahahahahahaha
2022-10-21 at 00:32
< report >At the bottom of the link page, this paragraph shows up 4 times "Title fields may be subject to user language preferences when authentication gets implemented later on. You can always fetch the full list of titles and apply your own selection algorithm."
#12 by blahblah35
2022-10-21 at 08:00
< report >And a little aboe we got "TODO: Footnotes with multiple references get duplicated. Pandoc is being weird. Need a workaround, because this will get annoying really fast. :(" :D It's a Pandoc issue here, it doesn't reuse footnotes, it just creates new ones.
#13 by ahahahahahahaha
2022-10-22 at 20:48
< report >Ah didn't see that!
#14 by ahahahahahahaha
2022-10-22 at 20:54
< report >Btw I'm curious if you have any stats on how heavily the tcp api is used, like how many tcp connections are established and how many queries does it answer per day?
#15 by Yorhel
2022-10-23 at 06:21
< report >About 6000 queries/day.
#16 by some-guy
2022-10-23 at 07:46
< report >Really appreciated! Doing requests and fetching the relevant content out of the response was rather awkward and error-prone, this will surely help with accessibility. No requests to add, just wanted to chime in for a thumbs up!
#17 by Yorhel
2022-10-24 at 16:15
< report >> Create a powerful API that supports fetching up to 100 entries per request.
> Specifically document how to efficiently fetch a large lists of identifiers.

And then there's still people who do it the inefficient way and use multiple IPs to bypass the rate limits. I'm impressed.
#18 by micah686
2022-10-26 at 00:19
< report >I'm glad you added a HTTPS API, since that should make things easier to work with over the TCP one.
#19 by aptx-4869
2022-10-27 at 16:12
< report >edit: <never mind I just didn't notice something>Last modified on 2022-10-27 at 16:18
#20 by aptx-4869
2022-10-27 at 21:04
< report >filter ulist on vn label?
#21 by micah686
2022-10-29 at 09:03
< report >With this new API, would it be possible to use tokens for authentication instead of username+ password? It would help with authentication being able to use a token as a replacement for their login info.
#22 by Yorhel
2022-10-29 at 10:17
< report >That's the plan, though I'm having doubts about the mobile client use-case. You wouldn't want to manually input a long token in some app. :/
#23 by blahblah35
2022-10-29 at 10:23
< report >I guess OAuth could be used here. Like twitter kinda.
#24 by micah686
2022-10-30 at 20:11
< report >I think it would also be helpful to add a changelog like with the TCP API, so people can see what's changed as new endpoints are added. Plus, it can help for developers to check if they've missed any new features.
#25 by bizuki
2022-11-05 at 09:45
< report >Thank you so much for this API. It will be much easier now to do something interesting with it.