Some love for the API

Posted in

#126 by tyr
2016-08-02 at 09:33
@onkelsam: It seems that VNDBUpdater wont get over 1325 visual novels when synchronizing my list. I don't know why because there isn't any error log. Before, it would add 25 vns every few seconds, but it stops at 1325. Waited 10 hours... nothing happened.
Deleted the local db and tried again... stops again at 1325.

@micah686: Ok, when I click on Login, nothing happens for ~10 seconds, then I see a huge list of vns on the right (I can see it's huge because the slider is very small), but the program immediately crashes with the same error again.
Well, that's progress I guess. Maybe your program didn't want to synch more than 1325 vns either? lol I don't know.

At this point I'm not sure if it's a problem with your tools, or the API or even my account. It would be nice if there was a log with vndb's server messages, so I can see what happened.
The throttling is also a real hassle. I don't want to wait hours every time I try to test something again. @yorhel, can't you give me some power user privileges or something like that, so that I don't have to wait hours every time I want to synchronize my list? ~.~
I feel that a list with my size is actually one which would really benefit from these tools, but at the same time the API's limits punish those who need this offline management the most.
#127 by onkelsam
2016-08-02 at 10:57
@tyr: I never tried it with so many VNs. I added a new release with tracing enabled, if you have the time and patience please download it ('VNDBUpdaterPortableWithTracing.zip') and send me the 'Eventlog.txt'.

link
#128 by tyr
2016-08-02 at 12:01
I hope it's ok that I uploaded the log to pastebin: link
It seems there is an error at the end while getting vns.
I wonder which vn produced this error? Maybe a deleted vn I still have in my list? Did you check for that?
#129 by onkelsam
2016-08-02 at 12:43
Thanks tyr.

You are perfectly right, that was the error. Didn't even think about checking that. I released a new version and added a check to that. It also has an improved file indexer.

link
#130 by tyr
2016-08-02 at 14:48
Oh yes, now it works!
I have another question: It seems my votes were not carried over. Every VN in my list has an "own score" of 0. Is this normal?
#131 by micah686
2016-08-02 at 15:24
@tyr, I'll look into it. And it takes a long time because I haven't implemented saving and reading to the sql database, which is what I am aiming for, so I can reduce issues like yours. Hopefully when I have that finished, it won't take that long, because it will only check for VNs that aren't on your list.

Or would you prefer an update list button, to manually update the list, but load from the offline database otherwise?

I'll try to get a new version to you in a day or two.
#132 by onkelsam
2016-08-02 at 15:37
@tyr: It should carry the votes over. Maybe try a restart (it syncs at every start, but it should be much faster). If that doesn't help could you please upload the eventlog again?
#133 by tyr
2016-08-02 at 16:28
@micah686: Thanks. I'm not sure I understand your question correctly though. But I think the "update list"-button is a good idea.

@onkelsam: Oh, indeed! After a restart my votes are there. But I'm pretty sure they weren't in the first session when I synched with my account for the first time.
Do you think it's possible to use floating-point numbers instead of integers for the votes? I'm reluctant to synchronize with VNDB again, I don't want 90% of my votes being accidentally overwritten with the rounded integer value ^^'' (Or will your program only update the entries I manually changed with VNDBUpdater?)
Can you also tell me what's the difference between Refreshing and Synchronizing? Is Refreshing getting the data from vndb and Synchronizing sending the data to vndb?
#134 by onkelsam
2016-08-02 at 16:40
It will only update votes you manually update with VNDBUpdater. Yes, I will add floating-point numbers in the next update (maybe today).

Refreshing: Gets the data for each visual novel new. E.g. the Screenshots/Characters/tags etc. will be updated. In the next update that will run as a Task in the background.

Syncing: Simply gets the data from VNDB.org of your VN-List/Wish-List and Vote-List and updates the local data. While syncing no local data is sent to VNDB.org.Last modified on 2016-08-02 at 20:29
#135 by onkelsam
2016-08-02 at 20:29
@tyr: I released a new version where the scores are displayed as floating-point and you can set custom scores (like 6.5). You can keep the old database, just overwrite all files except 'LocalVNStorage.rdb'.

Sending of data to VNDB.org only happens when you set the category/vote of a VN or add/remove a VN.
#136 by zolty
2016-08-02 at 22:36
@tyr
I don't think you have to worry too much about throttling even with a 1000+ item list, because you should only be getting those titles once (the get vn commands),
you can repeatedly fetch the entire userlist/wishlist/votelist (the get vnlist/wishlist/votelist commands) because it has a much smaller impact on throttling and get whatever titles are new which shouldn't be too many.

I set up an auto update for the titles in my lists (560+ titles) which refreshes (more like re-fetches) the lists every 2 days and I also refresh it manually as I am developing/debugging my app every once in a while and haven't been throttled in a few weeks I believe.

To other developers, I recommend that you backup your databases while debugging; for the releases, they should be saved in multiple different places in case they get accidentally deleted (e.g %localappdata% )
#137 by micah686
2016-08-07 at 06:45
Yorhel, would it be possible to add a way to login as another user in a Read Only mode, so only being able to view a user's vn list, wishlist, etc... without being able to change them? I think it would be very helpful feature for developers, so a developer could "log in" as another user and try to replicate an issue a user might be having.

My thought was you could login with the username and a null for the password for this.

If something like this already exists in the current API, could someone show me how to do it?
#138 by yorhel
2016-08-07 at 06:56
You can get the lists for other users by just changing the uid filter, e.g. "get votelist basic (uid=2)". This only works for users who have their lists set to public, but the same restriction will apply even if I'd add a "read only" mode.
#139 by zolty
2016-08-08 at 00:07
I've posted my project on github. link
You can download a release binary without having to compile. link
I don't want to hijack the thread so if you have any questions/issues/requests about it, make sure to read the "Project Goals.txt" before commenting,
I thought github would have a place where people can discuss projects but I guess not.
The project isn't completed and it isn't bug-free and a lot of documentation is still needed both in code and in the GUI.


@yorhel
Would it be possible for the 'get vnlist' etc commands to return an error if the list isn't public? Currently it just returns zero results, like someone with an empty list.
Furthermore, it would be really nice if 'Staff' was integrated into the API, seems like it could take a while but I would love to get results like Alpha.Last modified on 2016-08-08 at 00:16
#140 by entra
2016-08-09 at 18:42
@zolty I think it's missing SQLite.Interop.dll, it crashed right away until I added that to the project.

Looking pretty cool otherwise! Maybe open another thread for feedback?Last modified on 2016-08-09 at 18:43
#141 by zolty
2016-08-09 at 21:13
@entra I've made a new thread t8111.
I think that issue is to do with 32bit, make sure you build the program in 64bit only (link)
#142 by onkelsam
2016-08-12 at 22:28
@yorhel: Sorry to bother you again, but could you please include the height and width data for the image you get using the 'get vn details' (main image of the vn) command?

I'm currently having some problems as the DPI data of this image is often different and WPF scales the image wrong.

Thanks!
#143 by wakaranai
2016-08-12 at 23:02
WPF scales images because their DPI doesn't match resolution of your desktop. and this is not vndb's problem obviously. you could get native image width/height once you loaded it, you don't need vndb api for that.
#144 by onkelsam
2016-08-12 at 23:24
Yes it does and yes it isn't. Ok, sorry. I just thought as the rest of the screens provided by the API include width/height data it wouldn't be a big change and it would provide consistency in my program and in the api.
#145 by yorhel
2016-08-31 at 09:06
API update: there is a simple 'get user' command now.
#146 by zolty
2016-09-06 at 17:54
@yorhel
The relation field of vn relations objects should be described in the API page d11.
For example: ser = same series, preq = prequel, etc
#147 by yorhel
2016-09-06 at 18:02
Hmm, there's a few more fields like that; languages, platforms, anime types, media, producer types.

They're all defined in data/global.pl, and I'm not sure I want to manually duplicate all that on d11. Perhaps it's an idea to provide an automatically formatted list of those fields somewhere.
#148 by zolty
2016-09-06 at 18:22
Thanks for the link, perhaps just link to that page in d11 so other people don't ask the same thing.

Also, not a big deal to me but I don't think there's a way to distinguish official and unofficial relations with API.

Meanwhile, I really want to know if you're planning on having the Staff database accessible through the API.
#149 by yorhel
2016-09-06 at 18:27
I add stuff to the API whenever there's demand for it, I don't plan such features at all.

If you need the official flag for VN relations, that'd be easy to add. Adding staff is slightly more work, but that should be no problem either.
#150 by zolty
2016-09-06 at 22:45
I would appreciate if you added those things.

I'm now making vn screens show up on my program and I wonder if the last part of a screenshot's url is its unique identifier?
"s.vndb.org/sf/57/89357.jpg" (89357)

I was thinking of storing all the screens in the same folder but now that I think about it, that might get a bit chaotic so I might do as you do and store them in 100 folders with the last 2 parts of url.

Note: While I save cover images whenever a new VN is fetched, screenshots are only saved one VN at a time (sort of manually) so you shouldn't have to worry about bandwidth issues.Last modified on 2016-09-06 at 22:51

Reply

You must be logged in to reply to this thread.