Advanced search: (too) early testing & feedback
|#1 by yorhel|
2020-11-09 at 19:08
|< report >I've been working on an attempt to replace our current search & filter system with something fancier. The two primary problems with the current system is that it's neither very discoverable nor very flexible. I've often seen people asking if it's possible to perform certain searches that are, indeed, possible by clicking some fairly well hidden buttons, and similarly there have been many requests for search queries that are not possible with the current system and that can't easily be integrated into it.|
I have an alpha-quality version of a new advanced search running at /experimental/v. Feel free to give it a try and let me know what you think.
The first major difference is that the new advanced search has a few common filters directly visible and accessible, which hopefully remedies the discoverability problem. The second major change is that the advanced search allows you to combine filters to build custom nested queries. i.e. you can already say: Give me a Japanese VN that has an English Windows release but not a Japanese Windows release. Unfortunately, the UI for this is not as intuitive as I had hoped, so ideas to improve that are very welcome. EDIT: Should be much better now, let me know if things are still unclear.
The advanced search is still in a very early phase, so is missing a lot of filters and functionality. I'll at least add all filters that were available in the old system and hopefully a bunch more. EDIT: All old filters are implemented now.
This new advanced search will eventually replace the old filter system for all DB types: VNs, releases, staff and characters. I also plan to add a feature to load & save your queries, in addition to the old "save as default". Advanced search will also be added to tag & trait pages. And if I can get it working, to personal VN lists as well. Old URLs and saved filters will be converted to the new system automatically.
If there are any special search features you'd like to see, I'd love to hear about them and see if I can get it integrated. (Note, however, that even such a query builder approach is not nearly as flexible as raw SQL, so some things will remain impossible... Also, performance is going to be a challenge, as usual, but I'll try and see how far I can push our poor little server)
And to address a few points that rampaa brought up earlier:
-Age filter for charactersWill do.
-"Real sex" filter for characters
-VN search by anime relation (t3617.2413)
-"Visual Novel Filters" for Characters
-More powerful character filters (t12970.9)
-Exclusion/Inclusion with a minimum tag weight (just like AniDB)
-Developer filter for VNs (link)Already implemented, it's what I used to experiment with autocompletion-based filters.
-A resolution filter that doesn't rely on a static listHmm. The easy approach here is to have autocomplete-based selection like the developer filter, but that'll be pretty annoying if you want to search for things like "all 4:3 resolutions" or "larger than 1024x768". Needs more thought.
-"Has description" filter for VNs.This sounds like a search query that's only interesting for moderation tasks, for which I think SQL approach is generally more suited. I don't really object to such filters as long as I can find a way to have them not clutter the UI too much, because having a dropdown list with 100 possible filter types is not going to be very helpful. :/
Is there any chance of t12507.24 getting fixed?There's a good chance I can get that to work this time, yes.Last modified on 2020-12-14 at 15:04
|#2 by rampaa|
2020-11-10 at 06:03
|< report >The new advanced search looks really promising!|
Already implementedGreat! Though since "patch type" filter is not available in the new advanced search yet, we still can't get the same result we get from link.
that'll be pretty annoying if you want to search for things like "all 4:3 resolutions" or "larger than 1024x768"How about something like:
[Resolution: Free form with auto-completion support] [=|≤|≥] [Same aspect ratio: Checkbox]
This would allow queries like the following:
1024x768 = (☐) (Would only match with 1024x768)
1024x768 ≥ ☑ (Would match with 1024x768, 1600x1200 etc. but not with something like 1920x1080)
1024x768 ≥ ☐ (Would match with 1024x768, 1280x720, 1600x1200, 1920x1080 etc.)
1920x1080 ≤ ☐ (Would match with 1920x1080, 1024x768 etc. but not with something like 1600x1200)
1920x1080 ≤ ☑ (Would match with 1920x1080, 1280x720 etc. but not with something like 1024x768)
16x10 ≥ ☑ (Would match with all 16:10 resolutions)
4x3 ≥ ☑ (Would match with all 4:3 resolutions)
Since the aspect ratio checkbox is only relevant if ≥ or ≤ is selected, it can be hidden when = is selected to avoid confusion.
This sounds like a search query that's only interesting for moderation tasksActually I had a non-moderational use-case in mind. When I'm trying to look for what VN to read next through VNDB, getting VNs that lack description in the results is less helpful. So if possible, I would like to be able to filter those when I don't feel like visiting some other websites to read their synopses.
On a somewhat unrelated note, would it be possible to select a language filter by typing it in the new advanced search? As is, I need to scroll down to select Japanese, which is bothersome. :\Last modified on 2020-11-10 at 06:42
|#3 by surram|
2020-11-10 at 06:42
|< report >Love those ideas! More flexibility in searching is much appreciated!|
I do have one suggestion of a type of search I'd love to be able to do, though I'm not sure how feasible it is to implement this functionality. I often find myself wanting to run a search of the form 'find all VNs with properties A, B, and C that contain characters with traits X, Y, and Z'. For example, search for all VNs with an english translation that include a main character with blue hair.
|#4 by yorhel|
2020-11-15 at 08:54
|< report >Release date filter added. Once again a filter that required major changes to the data model and URL format. Here's hoping adding all the other filters will go smoother. -.-|
[Resolution: Free form with auto-completion support] [=|≤|≥] [Same aspect ratio: Checkbox]Looks like an excellent idea!
would it be possible to select a language filter by typing it in the new advanced search?I had the idea to do this, but it's slightly tricky to implement this well. The language and platform selection on the release edit form suffers from this, too. I'll experiment with a few approaches... later.
I often find myself wanting to run a search of the form 'find all VNs with properties A, B, and C that contain characters with traits X, Y, and Z'.That will absolutely be supported! Your example actually already works with the old system, but you'll hit the limits of that system pretty soon if you want to make it a little bit more advanced. The new search should be a lot more flexible.
|#5 by rampaa|
2020-11-20 at 16:51
|< report >Regarding the minimum tag weight selection, would it be possible to make the input field more flexible? |
My rationale is as follows: Since the default tag weight is 2, most people vote with that whether or not the tag plays actually a prominent role. So if one wants to be sure the tag plays a prominent role (I specifically plan to use this for exclusion purposes), they may want to search with tags with the minumum weight of 2.1 (or 2.5 etc.). But with the current options, we cannot exactly do that.
AniDB has 6 options for weight selection (+, *, *+, **, **+, ***), which I believe would be much better than having 3 options, but I think a more ideal solution would be allowing us to type between 0-3 (e.g. 2.1) in the minimum tag weight field.
|#6 by mrkew|
2020-11-20 at 17:49
|< report >#5|
1: The tag does apply to the visual novel, but is not too apparent or only plays a minor role.
2: The tag certainly applies to the visual novel.
3: The tag applies, is very apparent and plays a major role.
Where do you get that tag weight 2 is supposed to be prominent. It says right here that's it's supposed to be neither minor nor majorLast modified on 2020-11-20 at 17:50
|#7 by rampaa|
2020-11-20 at 18:11
|< report >#6: You completely misunderstood my point. I am saying that most people vote with the tag weight of 2 because it's the default tag weight, instead of properly selecting the tag weight. Say you have someone who voted with a 2 (because it's the selected tag weight by default) and someone else came along and correctly voted with a 3, what would be the average tag weight in that case? 2.5. And how do you search for something like 2.5 without including tags with the weight of 2? You cannot. The best you can do is either selecting 3 as the minimum tag weight and fail to get many instances where the tag plays a major role (because even a single "2" vote would make a tag weight non-3), or you can select 2 as the minimum tag weight and get many unwanted results. Thus my request.Last modified on 2020-11-20 at 22:45|
|#8 by yorhel|
2020-11-21 at 13:00
|< report >|
Regarding the minimum tag weight selection, would it be possible to make the input field more flexible?You're fast, had barely uploaded the feature and you found it already. And yes, as I was coding the weight selection it crossed my mind that the "3" selection is pretty useless and the others not really specific enough. I've kept the selection dropdown as I don't like manual inputs too much, but you can now select weights with increments of 0.2.
And I changed the URL format again. Hopefully that'll stabilize soon enough.
|#9 by rampaa|
2020-11-21 at 13:23
|< report >|
you can now select weights with increments of 0.2Nice!
Do you plan to implement parent-only search? (t12970.11)
|#10 by yorhel|
2020-11-21 at 13:38
|< report >|
Do you plan to implement parent-only search? (t12970.11)...if enough people want it, perhaps. Problem is that it would require maintaining an additional cache in the database, which I'm not sure is worth the hassle for a rarely used feature. Is parent-only search still needed?
|#11 by rampaa|
2020-11-21 at 13:46
|< report >|
Is parent-only search still needed?I can think of a few use cases for it but nothing I would use on a regular basis I guess. Though I'm guessing @Warfoki's input would be far more valuable than mine in this case.Last modified on 2020-11-21 at 13:51
|#12 by rampaa|
2020-11-24 at 20:46
|< report >I expected "Sex (spoiler)" filter to match with the real sex of the character even if it's not a spoiler but apparently it doesn't. So I tried to emulate the result I wanted by fiddling with the advanced search but I couldn't do it.|
The "where" clause of the following SQL is basically what I wanted to achieve: link
At first I tried to search like in the link but when I hit the search button it was normalized into link for some reason.
After that I tried the link and link but they didn't give me the expected results either.
So I have two questions to ask:
1) What did I do wrong?
2) Can the behavior of "Sex (spoiler)" filter be changed so that it always matches with the real sex of the character?Last modified on 2020-11-24 at 21:11
|#13 by yorhel|
2020-11-25 at 06:31
|< report >I was thinking yesterday how the and/or thing is super unintuitive the way it's displayed right now, thanks for confirming that. xD|
1) Working query with top-level or. EDIT: Better one. EDIT: Actually, neither works because excluding specific spoiler-sex values seems to exclude all characters that have no spoiler-sex. Hmm, probably a case of awkward NULL propagation.
2) Makes sense, I guess. EDIT: Done.Last modified on 2020-11-25 at 15:08
|#14 by rampaa|
2020-11-25 at 16:14
|< report >|
Done.Nice! On that note, wouldn't merging "Sex" and "Sex (spoiler)" filters make sense? I think a spoiler toggle like we have for "Tags" filter would suffice and that way "Add filter" UI for characters would be less cluttered.
|#15 by rampaa|
2020-11-27 at 19:14
|< report >Sometimes the UI makes it difficult to add different filter types. See the link for example. I can't seem to add a VN or release filter (without altering/removing the existing filters).|
|#16 by yorhel|
2020-11-28 at 09:32
|< report >|
Nice! On that note, wouldn't merging "Sex" and "Sex (spoiler)" filters make sense?That was my plan, but required more code and I just wanted to get this stuff out. May merge them later.
Sometimes the UI makes it difficult to add different filter typesOops, that's a bug. Fixed.
|#17 by rampaa|
2020-11-28 at 22:03
|< report >I'm getting an "Internal Server Error" from the following query: link|
Unrelated to the aforementioned error but there's something I want to confirm: link is being transformed into link when I hit the search button. Is this intentional? Since the language filter doesn't get the same treatment I couldn't be sure.Last modified on 2020-11-28 at 22:51
|#18 by yorhel|
2020-11-29 at 10:26
|< report >|
I'm getting an "Internal Server Error" from the following query: linkThe query is too slow and times out. I haven't optimized this at all yet, so it's not too surprising. Tag/trait exclusion queries are always going to be slow, but they should at least finish within a few seconds in the worst case.
|#19 by chibikko|
2020-11-29 at 14:58
|< report >Hi! This my first post here. A newcomer. How to search for VN with 2-3 same traits common in all or few characters, like - eyeglasses, singing, student council member ?|
|#20 by rampaa|
2020-11-29 at 15:41
|< report >Advanced search allows selecting non-searchable tags/traits but returns 0 results: link|
I think instead of making it not accept non-searchable tags/traits, you should just abolish the "not searchable" flag. See t3617.2320 for my reasoning.
(Also, in case you are interested in positive feedback: I think the current UI of the advanced search is way more intuitive than it was before. Thanks for your hard work!)
|#21 by yorhel|
2020-11-29 at 15:58
|< report >|
How to search for VN with 2-3 same traits common in all or few charactersNot currently possible... This might work if I'd add a "Has at least <number> characters that match these filters", but that'll get very complex and very slow. :(
I think instead of making it not accept non-searchable tags/traits, you should just abolish the "not searchable" flag.Meeeeh. That almost doubles the size of the 'tags_vn_inherit' cache and no doubt slows down tag searches even more. :(
I mean, it's possible if the demand is there, but how common is it to want to search for meta tags?
I think the current UI of the advanced search is way more intuitive than it was beforeThanks!
|#22 by rampaa|
2020-11-29 at 16:41
|< report >|
how common is it to want to search for meta tagsDepends on the tag/trait I guess. I wouldn't really use, say, Style for searching purposes personally, but I would totally use Animal Cosplay, Darker Sexual Contents, Gameplay Elements, Other Gameplay Elements, Half-breed, Sex Industry, Sexual Slavery, BDSM, Crime, Anal, Group Sex, Oral Sex, Empress, Royalty, Brother, Sister... The list goes on and on. :(
I mean, I should probably nag the mods about this, but listing every tag/trait that I think should be made searchable was a hard task to take (and there's always the chance of being ignored :() so I didn't. Abolishing the flag itself seemed like an easier and cleaner solution (because honestly, making most of the non-searchable traits searchable would make so much sense, please have a look at it yourself: link). But I guess if abolishing the flag would cause a performance hit, I should just nag the mods after all. :(Last modified on 2020-11-29 at 16:44
|#23 by yorhel|
2020-11-29 at 16:48
|< report >Of course, marking all tags as searchable comes with the same performance overhead. I'll do some testing once I get to optimizing the queries, which I definitely need to do anyway.|
|#24 by rampaa|
2020-11-29 at 16:55
|< report >|
marking all tags as searchable comes with the same performance overheadI thought making something like Theme searchable would be way more costly than than all the tags/traits I've listed combined, did I assume wrong?
|#25 by yorhel|
2020-11-29 at 17:10
|< report >That's probably correct, at least in terms of number of rows cached. But I don't really know yet how that translates into query performance - if I can fix the queries to use indices efficiently then the row count wouldn't even matter much and we can simply make everything searchable. If I can't, we'll indeed need to be selective in marking tags/traits as searchable.|