Some love for the API

Posted in

#176 by kurothing
2017-06-23 at 23:08
< report >Documentation for "get release" says the field is "animated" while the release logs (and actual response) is "animation" o.o
#177 by yorhel
2017-06-24 at 04:45
< report >@kurothing: Oops. Fixed the doc.
#178 by kurothing
2017-08-05 at 07:46
< report >The documentation states for the throttled error...
You have used too many server resources within a short time, and need to wait a bit before sending the next command. The type of throttle is given in the "type" member, and the "minwait" and "fullwait" members tell you how long you need to wait before sending the next command and when you can start bursting again (this is the recommended waiting time), respectively. Both values are in seconds, with one decimal after the point.

Is the wording of the last sentence supposed to imply that the result will always have a value after the decimal? Because it doesn't have one if the value is a full number.
#179 by yorhel
2017-08-05 at 07:53
< report >Uh, that wording is bad. Full numbers will just be formatted as integers. Should be fixed now.

(The formatting of numbers *should* not matter, but I realize that not all languages and JSON libraries handle this the same way, which is unfortunate)
#180 by kurothing
2017-08-05 at 12:37
< report >Sadly my library only checked for decimals when running through a converter, so when a full integer was given it became a bit awkward. (Anyone want to wait 2016 years, 7 months and give or take 5 days between commands? :D)Last modified on 2017-08-05 at 12:37
#181 by micah686
2017-08-14 at 06:10
< report >Yorhel, would it be possible to add an "id" filter to the "get votelist", "get vnlist", and "get wishlist" commands, for filtering by the Visual Novel ID?

I'm aware the command also returns the Visual Novel ID. However, if I am trying to find the vote and added for the votelist for a specific VN that a specific user voted on, I have to loop through every single entry until I come across that specific VN.
This isn't normally an issue, but for users like tyr, it requires looping through hundreds of entries in order to find one specific entry.

However, if an "id" filter was available, I could find the specific entry out of a large amount of entries, possibly hundreds or thousands.
#182 by yorhel
2017-08-14 at 16:26
< report >@181: Done (+ a little bit more, see changelog).
#183 by micah686
2017-08-14 at 18:41
< report >Thanks.
Also, on the changelog, shouldn't it be added 'vid' instead of uid?

"Add 'uid' field to "get votelist/vnlist/wishlist" commands"
#184 by yorhel
2017-08-14 at 19:44
< report >Nope, I did add a "uid" field.
#185 by micah686
2017-08-17 at 00:31
< report >The 'vid' filter doesn't seem to be working. If I try something like: "get votelist basic (vid=4)", I get an error message: "Unknown Field 'vid'.

However, if I try "get votelist basic (vn=4)", I get the proper response:
"results {\"items\":[{\"uid\":8,\"added\":1196396808,\"vote\":90,\"vn\":4},....."
#186 by yorhel
2017-08-17 at 05:34
< report >Ah yes, the doc was wrong, should indeed be "vn" instead of "vid".
#187 by micah686
2017-11-10 at 06:02
< report >So I've finally finished the first version of my visual novel manager v2. I've worked hard to try to keep it error and bugfree as possible. You can get it from If anyone encounters a bug, please report it here:

I'd also like to thank kurothing and onkelsam for their code on github. I wouldn't have been able to complete it without VndbSharp and the trait code from VndbUpdaterLast modified on 2017-11-10 at 06:03
#188 by zolty
2018-01-16 at 21:55
< report >@yorhel

What kind of command would one use to get visual novels that don't have a proper release date (usually it's displayed as tba)?

It seems this command returns those (along with others that match the filter):
get vn basic (released > "2017" and released <= "9999") {"results":25, "page":18}

Is there a way to just get ones without any date?
Can i trust that previous command to get every vn released from 2018 onwards or without a date?

Also, just noticed that a value of null for released is different than tba, the command above returns vns with tba but I don't think it returns those with null.Last modified on 2018-01-16 at 22:02
#189 by yorhel
2018-01-17 at 17:40
< report >@zolty: You should be able to do (released=null) to get VNs for which no release date is known, and (released="tba") for TBA (equivalent to (released=99999999), because thats how it is represented internally).
#190 by zolty
2018-01-17 at 18:02
< report >Great, thanks.
#191 by kurothing
2018-01-20 at 03:46
< report >I would like to request an additional login method that permits sending the password that is encrypted via a public/private key pair.

My primary goal in having such a method is to reduce the problems that exist when trying to use VndbSharp for authentication. Because the way that strings are handled in the .Net Framework, they can hang around for an undefined period with no way to easily clear them, and the only option to fix that is to dip into managed memory, which still doesn't resolve the issue at hand... The password is being stored in memory in plain text.

Even if you do that, when creating the json for the login request, it'll still end up in memory, in plain text... It's just an outright security flaw, and i cannot think of a solution that only requires modifying VndbSharp...

So after thinking about it for a long time, the only solution i can come up with that is secure, is to have the API accept some sort of encrypted password, preferably something secure and generic like RSA. I came to this conclusion, because beyond the initial login (For the application), the only string that could be stolen would be encrypted password, and without the private key it would be impossible to decrypt it, meaning that the password is safe even when transported over an insecure socket or in a memory dump.

I understand that this change could cause performance issues in regards to logging in, since every login request would require decrypting the password to verify it, then hashing it... But beyond setting up something akin to an OAuth flow, which has tokens in every request, which would still have the same memory dump problem as far as VndbSharp and .Net goes but less problematic since with proper permissions / claims the token could be set to read only, i cannot think of a solution.

Ultimately, i understand if you say no to this request, and i'll keep VndbSharps user login methods behind custom builds (Requiring the developer to use a preprocessor to get the features, and editing a bit of the library) to discourage usage of them, since they are simply put...insecure, despite enforcing secure connections.
#192 by yorhel
2018-01-20 at 06:55
< report >@kurothing: Uhm, what is your threat model? How do you estimate the likelyhood that an attacker will use that approach to grab someone's password?

You do realize that sensitive information is all over RAM, right? Password managers and full disk encryption methods keep the decryption key in memory for as long as your vault/disk remains open. Passwords go through your browser when logging in to a site, also in memory. Sure, some (but definitely not all) of these systems put some effort into making sure the password isn't paged out to disk or attempt to wipe it from memory after use, but those techniques are just stopgap measures. It's good to be paranoid (I mean, I am), but it's always good to keep the greater context in mind.

In any case, I don't think public key crypto is the solution you'd want. API keys or session tokens may be a better solution. (But session tokens aren't very useful if you can keep the TCP connection alive).

EDIT: If hostile account takeover is your concern, there's already a MUCH easier approach: Just read the session token from the browser's cookie store. Which is unencrypted and stored on disk.Last modified on 2018-01-20 at 08:09
#193 by kurothing
2018-01-20 at 21:37
< report >With spectre and meltdown, that memory can now be read from anywhere, even your browser, if they are not patched...and are we really going to rely on end users patching their systems?

And while browsers and other applications surely store passwords in memory via plain text, they also most likely don't have the problem of not being able to clear that memory easily and reliably like the problem that exists in the .Net world, which is amplified because the password will need to be kept in memory the entire time the application is running to accommodate the user experience of one login per session for applications. Even if stored encrypted, there is the Json payload as well :(

As far as hostile takeovers, a session cookie is already rather limited takeover in the grand scheme of things, a huge portion of the population use the same or similar passwords for multiple sites, with the recently found vulnable and the admitly poor handling of strings in .Net that is a terrible combination.

As far as why I went for encryption over tokens, it was because I couldn't think of a smooth way to authorise tokens for the end user over a TCP API :P maybe I'm still stuck in the concept of RESTful APIs?
#194 by yorhel
2018-01-21 at 07:56
< report >
and are we really going to rely on end users patching their systems?
Unfortunately, that's the only sane solution. Meltdown and Spectre have already caused enough wreckage and instability at the OS level, now imagine what would happen if *every* application then also applied its own mitigations (and half-assed ones at that, because the problem is impossible to properly mitigate at the application level, apart from limiting arbitrary code execution and adding process separation where that makes sense).

If your thread model is "password extraction from memory" combined with "no way to properly clear strings in .Net", then solutions like encryption or session tokens will only decrease the likelyhood of a successful extraction by a tiny bit, since the password will enter the application one way or another if the user intents to log in. The only real mitigation then is to use API keys, which will reduce the impact of a successful extraction.

Considering the unlikelyhood that someone will use this method to extract a password from some obscure website before the user's systems are patched - especially when there are thousands of similar and higher-profile targets to attack in the same manner - I'm not convinced that it is worth mitigating.
#195 by vndbandroid
2018-01-21 at 21:18
< report >Hey hey.
@yorhel I've been playing with the API lately and something looks strange to me: the "login" command takes 800-1000 ms to return a response on my app (measured once the socket is already connected). It feels a bit long, considering that the SQL query should be indexed on the username, and the actual socket connection takes ~200 ms and a basic "get" command ~200-300 ms.
Is there a bottleneck that I am not aware of in this command, or is there per chance an issue?Last modified on 2018-01-21 at 21:20
#196 by yorhel
2018-01-21 at 21:59
< report >@vndbandroid: Yeah, that's the time it takes to hash the password. I may have been a bit too paranoid when setting the scrypt parameters.
#197 by kurothing
2018-01-23 at 07:43
< report >
Considering the unlikelyhood that someone will use this method to extract a password from some obscure website before the user's systems are patched - especially when there are thousands of similar and higher-profile targets to attack in the same manner - I'm not convinced that it is worth mitigating.
I can't say i'm too fond of this thinking, especially since it encourages the single point of failure concept, allowing users to feel secure if the big targets are safe, while in the mean time the small guys are vulnerable, because when this is combined with what i mentioned about users using passwords across multiple sites, it becomes a big problem. However, i can see that you're not willing to budge in that regard.

So as a compromise....since you seemingly appear to be okay with using API keys (Based off of your response, and preference of mentioning them), could we possibly see that being added to the site? (It'll require more work then just adjusting the API, since it will require a location to generate, re-generate (Key has been exposed for example), or delete the keys on the site as well, and possibly more if you want to get into permissions for the key...)

In my opinion, API keys aren't too far different from passwords, they are just more restrictive and offer a smaller attack vector (Can only modify the users data at worst, rather than stealing the entire account which is far better in the grand scheme of things, admittedly) but at this point, anything is better then raw passwords.
#198 by yorhel
2018-01-23 at 08:05
< report >
about users using passwords across multiple sites
There's a limit to how many technical solutions you can implement to work around insecure human behavior. It's certainly good to have some measures to protect against bad use of passwords, but that doesn't mean we should do everything we possibly can to protect people who don't care about security (until it's too late).

And in any case, security isn't binary, adding more layers of security is always a matter of tradeoffs. In this case:

- Encrypting passwords: Increases the attack surface by adding more (and more complex) code. By definition, more code = more bugs, and I've seen far too many custom encryption schemes (that includes using otherwise sane methods such as AES/RSA) that ended up being completely insecure due to a tiny oversight or implementation bug.

- Sessions: This might work, but in order to obtain a session key your application will need to authenticate in *some* way. And if that "some way" is sending a password over the network, the password will still enter your application at some point.

- API keys: As you mentioned, this adds some complexity to the API and the website, so more work for me :P. It's also more annoying and less intuitive for users, in that they have a key to find on the website and copy-paste into the application.

It's not so much that I have a preference for API keys as that I have little faith that the other solutions will do anything. I'm open to be convinced otherwise. :)

EDIT: To connect these arguments to my previous ones about likelyhood: It's perfectly fine to implement measures if there's a significant risk of the compromise happening. It makes less sense to accept the tradeoffs when the threat model is "not an easy to execute attack targeted at a single user on a particular application". And still less sense when a user can already protect himself by applying patches and not reusing passwords.Last modified on 2018-01-23 at 08:18
#199 by micah686
2018-03-22 at 06:48
< report >This may not be the right section, but it looks like you replaced the old release icons with updated SVG icons(Or I've not been observant). Are you planning to update the rest of the site icons with newer SVGs as well?

I think they look better than the old ones btw.
#200 by yorhel
2018-03-22 at 07:48
< report >No icons have been replaced, but new icons have been added. And these happen to be SVG. The other icons will likely be replaced with SVG eventually, most likely during the overall redesign.

Of course, if you have a nice set of SVG icons I'd be happy to replace them earlier. The main problem is that finding or designing a consistent set of icons is not an easy task and takes some time.