Some love for the API

Posted in

#201 by kurothing
2018-05-12 at 12:46
< report >It appears when you log in via the API, the username must be transformed to lowercase before being sent...

Can this be confirmed? And if so, could we possibly get something server side to do that transformation for us in the API so that issues like this don't occur.
#202 by yorhel
2018-05-12 at 13:10
< report >Ah, that's a bug. I wasn't aware that the web login form accepted uppercase characters in the username, but it does, and obviously the API should be consistent with that.

#203 by kurothing
2018-05-13 at 23:56
< report >On the API Documentation page, it states....

Design overview
TCP-based, all communication between the client and the server is done using one TCP connection. This connection stays alive until it is explicitely closed by either the client or the server.

Does this imply that the connection to Vndb should remain opened for as long as possible, or should it be closed as soon as it's finished being used? Or does it not matter and it's up to developer preference?
#204 by yorhel
2018-05-14 at 04:56
< report >
Does this imply that the connection to Vndb should remain opened for as long as possible,
Whatever makes the most sense for your application.
#205 by micah686
2018-06-03 at 19:11
< report >With work being done on a site redesign (, will that also include any changes/rework of the API at all?
#206 by yorhel
2018-06-03 at 19:16
< report >The API and the site share very little code, so I have nothing planned in that regard at the moment. Do you want a reworked API or were you just asking out of curiosity?
#207 by micah686
2018-06-08 at 23:40
< report >A little bit of both. There are a few sections of the API that may need some work. Also, even though I know you said that the API and the site share very little code, I figured right after a site redesign like what is planned would be a good time as any to rework the API.
#208 by mooncake4132
2018-06-13 at 04:54
< report >It seems like the "instance of" information for the characters is currently unavailable in the APIs. Is there a plan to make that happen?
#209 by yorhel
2018-06-13 at 16:07
< report >@208: Added.
#210 by mooncake4132
2018-06-13 at 16:42
< report >Wow. Thank you. Didn't expect it to be added that quickly.
#211 by hitchiker42
2018-07-21 at 01:23
< report >Would it be possible to add the revision number and/or the timestamp of the last modification as a field/filter? I cache results from api queries, but as it currently stands there is no way to check if the information I have cached is out of date. Currently I just set a limit on my end, but this can easily result in out of data data and unnecessary updates on my end. I'm not sure how many people actually use the api so it might not be an issue, but I imagine adding something like this could reduce server load by reducing the amount of data transferred.

I've looked through the code/database schema and it doesn't seem like it would be too difficult to implement this, though it does present some issues (like how exactly a filter for the revision number would work.) If you have a better way to solve this problem or if there's a way with the current api that I'm just missing that would be fine too.
#212 by micah686
2018-07-27 at 04:30
< report >I would appreciate this as well, since it would help with determining to use cached data, or live data from the API
#213 by rampaa
2018-08-05 at 08:50
< report >It would be great if we could specify (minimum) tag scores when filtering VNs by their tags.
#214 by rampaa
2018-08-06 at 09:41
< report >Is there a way to get child tags of a tag through the API?

Also (released>"2005" and released<"2018") works as intended but (released>"2005-1-1" and released<"2018-1-1") returns nothing. Am I missing something obvious?Last modified on 2018-08-06 at 09:45
#215 by yorhel
2018-08-06 at 11:45
< report >
but (released>"2005-1-1" and released<"2018-1-1") returns nothing
It should return an error, according to link - "2005-01-01" ought to work though, format is YYYY-MM-DD.
#216 by micah686
2019-05-17 at 23:57
< report >Yorhel, since you are updating the password policy of VNDB, will you also be updating the API to switch from using for passwords to something more secure?
#217 by roadi
2019-05-27 at 23:40
< report >As far as I was able to gather, there does not currently seem to be a way to filter characters based on their birthdays.
Could this be added?

For example, the following syntax might make sense ('0' would signify "don't care"):
get character basic (birthday = [ 12, 31 ])
get character basic (birthday = [ 12, 0 ])

Or perhaps as separated filters:
get character basic (b_day = 31)
get character basic (b_month = 12)

The web interface could perhaps be augmented likewise.
#218 by yorhel
2019-09-20 at 07:00
< report >Continuing from t12882.10 to draft some ideas for a new HTTP-based API. Before people start to worry about the fate of the existing API: It will remain available and will continue to be maintained for as long as it has users. Which will be quite a while.

Some API ideas/requirements:
- Feature parity with the existing API would be nice, but not a hard requirement. Certainly not for a first version, can always start simple and expand later on.
- Main use cases are: advanced querying, fetching/synchronizing database entries and list management.
- Excluded use cases: Database editing and anything discussion board related.
- Consistently use character+number identifiers (e.g. "v5") instead of numeric identifiers. The website has always been consistent in doing this, but I should have copied the idea over to the dumps and the APIs as well. Can't retroactively change that now. Unambiguous identifiers may provide an opportunity to consolidate or clean up some API aspects. Or not.
- For list management purposes: Session-based authentication.
- Optional: Support for "give me a list of entries changed since $x" or "give me this list of entries but only those that have changed since $x" would be nice for data synchronisation use cases. $x could either be a timestamp or (more reliable) a global incrementing change identifier (which the DB happens to have, internally:

Implementation ideas/requirements:
- Declarative, or at least concise, descrption of the supported paths, parameters, filters, outputs, SQL and anything that depends on the database schema or that may be extended later on. The current API has something like that, but it doesn't have to be *that* concise or magical. Could be implemented as Rust macros, derived structs, a static data structure or a separate file loaded at run-time, doesn't really matter, I just hate repeating myself. :P Bonus points if API documentation can be generated from that description.
- I'm not sure how advanced querying should be exposed in the API, but if the current filter strings are going to be reused, I happen to have an old parser that might be useful.
- The server is going to be behind a reverse proxy, HTTPS support isn't very useful but it should understand the X-Forwarded-For header.
#219 by roadi
2019-09-20 at 11:08
< report >@#218:
I'm currently looking into using Diesel for interfacing with the SQL database.
This should fit (does it?) the use-case quite well (assuming that you do not hate ORMs with a passion).

The general structure could perhaps be divided quite cleanly into two parts: the backend DB client and the frontend server, where the former would also be usable as a standalone program with a local DB.
#220 by yorhel
2019-09-20 at 11:31
< report >
assuming that you do not hate ORMs with a passion
I do, but Diesel is also (or primarily?) a generic type-safe query builder, and I *do* like those. You're going to be building some complex queries for the API, if Diesel's query builder provides enough flexibility to do so, it's a good choice. Just be aware of potential repetition: If you have to duplicate the database schema in Rust, you're doing something wrong. But if my memory serves me, Diesel has a code generator for that.

backend DB client
This can work, but be aware of overengineering and (again) repetition (I'm being repetitive..). In my view, the best "database client" is plain old SQL, but I don't mind to be proven wrong.
#221 by wilk220
2019-10-03 at 15:45
< report >One advantage of the API being HTTP-based is a possibility of reusing existing page URLs for API endpoints. Danbooru and Reddit return data in different formats depending on the extension appended to the URL (e.g. /posts.json, /posts.atom for /posts; /r/all.json, /r/all.rss for /r/all), but it could also be implemented by checking the Accept HTTP header and returning the response in a fitting format. Obviously, this has some disadvantages. It could complicate web server setup if the API server was a separate application (could be mitigated by using a URL prefix or a different domain maybe?). I'm guessing the aforementioned sites just implement their read-only part of the API as different views for the same data, one of which is HTML. This makes sense to me, as (at least in theory) it means there aren't multiple reimplementations of almost the same thing, the code is being shared, and both views (HTML and something more machine-readable) have the same semantics, including the way they accept arguments, handle pagination, etc. (That last point is what's bothering me a bit with the current TCP API, the website and the API have completely different mutually unintelligible ways of filtering.) Some API options such as "give me only the 'id' and 'title' fields" might not make sense for the HTML views, though. There might be other points which could make this a bad design decision, I'm just throwing this out since I really like the consistency it provides, e.g. if I had a VN search URL with a lot of filters set and wanted to do something with the results, I could just add ".json" to it, prepend "/api/" to it or whatever else and get a machine-readable response without even reading the docs.Last modified on 2019-10-03 at 16:29
#222 by yorhel
2019-10-03 at 16:23
< report >I agree it would be convenient and easy to use (for some use cases, at least), but my initial reaction is that it wouldn't make a good API, on various points:

- I'd rather not tie UI and API together for stability reasons: The UI may change (both the URLs and what is being displayed on a page), but such changes really should not be reflected in the API, for compatibility reasons.
- The UI isn't a good database API. The filters are a good example of this: the TCP API filters are in some ways more powerful because they aren't limited to whatever I manage to create a friendly UI for (they are in other ways less powerful as well, but that's not a limitation of their design. As opposed to the UI filters which were *designed* to be unstable and inflexible).
- The UI has duplicated functionality in various places. For example, listings and entry pages are essentially the same thing from an API perspective. I'm not interested in maintaining an API with such duplication.

IMO, a good API is one that reflects the data model. The UI may deviate from it here and there.
#223 by wilk220
2019-10-03 at 16:38
< report >
The UI may change (both the URLs and what is being displayed on a page)

I'd say breaking the URLs is bad regardless of whether it's API URLs or webpage URLs, since you can never be sure if there isn't anything linking to a given page. If the resource still exists, broken links introduce unnecessary friction and confusion to the outside visitors.

You do raise good points, though. In case of Danbooru there are multiple endpoints which essentially do the same thing (e.g. /posts and /posts/:id), and I do agree the filters used on the website seem quite unwieldy for hand-crafting (can't really comment on their flexibility since I haven't played with them enough).
#224 by yorhel
2019-10-03 at 16:57
< report >
I'd say breaking the URLs is bad regardless of whether it's API URLs or webpage URLs
I agree, and the VNDB codebase does have redirects for various old URLs and parameters. But there's a difference: These are provided on a best-effort basis, and I do intentionally break things when I feel it's not worth the effort. The scope of the UI is simply way too large for me to provide any guarantees about its stability. An API can have a much more limited scope and as such ought to be easier to maintain with backwards compatibility guarantees. I don't think I've ever broken a thing in the TCP API, for example, despite it being pretty old.

One thing that I'm seriously afraid of is comitting to stability with such a large scope that it becomes impossible for a single person to extend/improve the site. VNDB development is still pretty much a one-man effort*, and there's only so much I can do.
(* with the occassional drive-by contributions, sure, but that doesn't help much with maintenance).
#225 by micah686
2020-09-07 at 22:40
< report >Yorhel, a concern I have about a change to the API. I noticed that Wikipedia links were deprecated in favor of Wikidata. My concerns are that there is not as much relevant information on Wikidata than there is on Wikipedia. For example: vs I'm concerned that this deprecation will give less relvant data.

Also, would you be willing to add the Store Page for VnInfo to the Api?