Advanced search: (too) early testing & feedback

Posted in

#1 by yorhel
2020-11-09 at 19:08
< report >I've been working on an attempt to replace our current search & filter system with something fancier. The two primary problems with the current system is that it's neither very discoverable nor very flexible. I've often seen people asking if it's possible to perform certain searches that are, indeed, possible by clicking some fairly well hidden buttons, and similarly there have been many requests for search queries that are not possible with the current system and that can't easily be integrated into it.

I have an alpha-quality version of a new advanced search running at /experimental/v. Feel free to give it a try and let me know what you think.

The first major difference is that the new advanced search has a few common filters directly visible and accessible, which hopefully remedies the discoverability problem. The second major change is that the advanced search allows you to combine filters to build custom nested queries. i.e. you can already say: Give me a Japanese VN that has an English Windows release but not a Japanese Windows release. Unfortunately, the UI for this is not as intuitive as I had hoped, so ideas to improve that are very welcome.

The advanced search is still in a very early phase, so is missing a lot of filters and functionality. I'll at least add all filters that were available in the old system and hopefully a bunch more.

This new advanced search will eventually replace the old filter system for all DB types: VNs, releases, staff and characters. I also plan to add a feature to load & save your queries, in addition to the old "save as default". Advanced search will also be added to tag & trait pages. And if I can get it working, to personal VN lists as well. Old URLs and saved filters will be converted to the new system automatically.

If there are any special search features you'd like to see, I'd love to hear about them and see if I can get it integrated. (Note, however, that even such a query builder approach is not nearly as flexible as raw SQL, so some things will remain impossible... Also, performance is going to be a challenge, as usual, but I'll try and see how far I can push our poor little server)

And to address a few points that rampaa brought up earlier:
-Age filter for characters
-"Real sex" filter for characters
-VN search by anime relation (t3617.2413)
-"Visual Novel Filters" for Characters
-More powerful character filters (t12970.9)
-Exclusion/Inclusion with a minimum tag weight (just like AniDB)
Will do.

-Developer filter for VNs (link)
Already implemented, it's what I used to experiment with autocompletion-based filters.

-A resolution filter that doesn't rely on a static list
Hmm. The easy approach here is to have autocomplete-based selection like the developer filter, but that'll be pretty annoying if you want to search for things like "all 4:3 resolutions" or "larger than 1024x768". Needs more thought.

-"Has description" filter for VNs.
This sounds like a search query that's only interesting for moderation tasks, for which I think SQL approach is generally more suited. I don't really object to such filters as long as I can find a way to have them not clutter the UI too much, because having a dropdown list with 100 possible filter types is not going to be very helpful. :/

Is there any chance of t12507.24 getting fixed?
There's a good chance I can get that to work this time, yes.Last modified on 2020-11-15 at 07:56
#2 by rampaa
2020-11-10 at 06:03
< report >The new advanced search looks really promising!

Already implemented
Great! Though since "patch type" filter is not available in the new advanced search yet, we still can't get the same result we get from link.

that'll be pretty annoying if you want to search for things like "all 4:3 resolutions" or "larger than 1024x768"
How about something like:
[Resolution: Free form with auto-completion support] [=|≤|≥] [Same aspect ratio: Checkbox]

This would allow queries like the following:
1024x768 = (☐) (Would only match with 1024x768)
1024x768 ≥ ☑ (Would match with 1024x768, 1600x1200 etc. but not with something like 1920x1080)
1024x768 ≥ ☐ (Would match with 1024x768, 1280x720, 1600x1200, 1920x1080 etc.)
1920x1080 ≤ ☐ (Would match with 1920x1080, 1024x768 etc. but not with something like 1600x1200)
1920x1080 ≤ ☑ (Would match with 1920x1080, 1280x720 etc. but not with something like 1024x768)
16x10 ≥ ☑ (Would match with all 16:10 resolutions)
4x3 ≥ ☑ (Would match with all 4:3 resolutions)

Since the aspect ratio checkbox is only relevant if ≥ or ≤ is selected, it can be hidden when = is selected to avoid confusion.

This sounds like a search query that's only interesting for moderation tasks
Actually I had a non-moderational use-case in mind. When I'm trying to look for what VN to read next through VNDB, getting VNs that lack description in the results is less helpful. So if possible, I would like to be able to filter those when I don't feel like visiting some other websites to read their synopses.


On a somewhat unrelated note, would it be possible to select a language filter by typing it in the new advanced search? As is, I need to scroll down to select Japanese, which is bothersome. :\Last modified on 2020-11-10 at 06:42
#3 by surram
2020-11-10 at 06:42
< report >Love those ideas! More flexibility in searching is much appreciated!

I do have one suggestion of a type of search I'd love to be able to do, though I'm not sure how feasible it is to implement this functionality. I often find myself wanting to run a search of the form 'find all VNs with properties A, B, and C that contain characters with traits X, Y, and Z'. For example, search for all VNs with an english translation that include a main character with blue hair.
#4 by yorhel
2020-11-15 at 08:54
< report >Release date filter added. Once again a filter that required major changes to the data model and URL format. Here's hoping adding all the other filters will go smoother. -.-

[Resolution: Free form with auto-completion support] [=|≤|≥] [Same aspect ratio: Checkbox]
Looks like an excellent idea!

would it be possible to select a language filter by typing it in the new advanced search?
I had the idea to do this, but it's slightly tricky to implement this well. The language and platform selection on the release edit form suffers from this, too. I'll experiment with a few approaches... later.

I often find myself wanting to run a search of the form 'find all VNs with properties A, B, and C that contain characters with traits X, Y, and Z'.
That will absolutely be supported! Your example actually already works with the old system, but you'll hit the limits of that system pretty soon if you want to make it a little bit more advanced. The new search should be a lot more flexible.
#5 by rampaa
2020-11-20 at 16:51
< report >Regarding the minimum tag weight selection, would it be possible to make the input field more flexible?

My rationale is as follows: Since the default tag weight is 2, most people vote with that whether or not the tag plays actually a prominent role. So if one wants to be sure the tag plays a prominent role (I specifically plan to use this for exclusion purposes), they may want to search with tags with the minumum weight of 2.1 (or 2.5 etc.). But with the current options, we cannot exactly do that.

AniDB has 6 options for weight selection (+, *, *+, **, **+, ***), which I believe would be much better than having 3 options, but I think a more ideal solution would be allowing us to type between 0-3 (e.g. 2.1) in the minimum tag weight field.
#6 by mrkew
2020-11-20 at 17:49
< report >#5
1: The tag does apply to the visual novel, but is not too apparent or only plays a minor role.
2: The tag certainly applies to the visual novel.
3: The tag applies, is very apparent and plays a major role.

Where do you get that tag weight 2 is supposed to be prominent. It says right here that's it's supposed to be neither minor nor majorLast modified on 2020-11-20 at 17:50
#7 by rampaa
2020-11-20 at 18:11
< report >#6: You completely misunderstood my point. I am saying that most people vote with the tag weight of 2 because it's the default tag weight, instead of properly selecting the tag weight. Say you have someone who voted with a 2 (because it's the selected tag weight by default) and someone else came along and correctly voted with a 3, what would be the average tag weight in that case? 2.5. And how do you search for something like 2.5 without including tags with the weight of 2? You cannot. The best you can do is either selecting 3 as the minimum tag weight and fail to get many instances where the tag plays a major role (because even a single "2" vote would make a tag weight non-3), or you can select 2 as the minimum tag weight and get many unwanted results. Thus my request.Last modified on 2020-11-20 at 22:45
#8 by yorhel
2020-11-21 at 13:00
< report >
Regarding the minimum tag weight selection, would it be possible to make the input field more flexible?
You're fast, had barely uploaded the feature and you found it already. And yes, as I was coding the weight selection it crossed my mind that the "3" selection is pretty useless and the others not really specific enough. I've kept the selection dropdown as I don't like manual inputs too much, but you can now select weights with increments of 0.2.

And I changed the URL format again. Hopefully that'll stabilize soon enough.
#9 by rampaa
2020-11-21 at 13:23
< report >
you can now select weights with increments of 0.2

Do you plan to implement parent-only search? (t12970.11)
#10 by yorhel
2020-11-21 at 13:38
< report >
Do you plan to implement parent-only search? (t12970.11)
...if enough people want it, perhaps. Problem is that it would require maintaining an additional cache in the database, which I'm not sure is worth the hassle for a rarely used feature. Is parent-only search still needed?
#11 by rampaa
2020-11-21 at 13:46
< report >
Is parent-only search still needed?
I can think of a few use cases for it but nothing I would use on a regular basis I guess. Though I'm guessing @Warfoki's input would be far more valuable than mine in this case.Last modified on 2020-11-21 at 13:51
#12 by rampaa
2020-11-24 at 20:46
< report >I expected "Sex (spoiler)" filter to match with the real sex of the character even if it's not a spoiler but apparently it doesn't. So I tried to emulate the result I wanted by fiddling with the advanced search but I couldn't do it.

The "where" clause of the following SQL is basically what I wanted to achieve: link

At first I tried to search like in the link but when I hit the search button it was normalized into link for some reason.

After that I tried the link and link but they didn't give me the expected results either.

So I have two questions to ask:
1) What did I do wrong?
2) Can the behavior of "Sex (spoiler)" filter be changed so that it always matches with the real sex of the character?Last modified on 2020-11-24 at 21:11
#13 by yorhel
2020-11-25 at 06:31
< report >I was thinking yesterday how the and/or thing is super unintuitive the way it's displayed right now, thanks for confirming that. xD

1) Working query with top-level or. EDIT: Better one. EDIT: Actually, neither works because excluding specific spoiler-sex values seems to exclude all characters that have no spoiler-sex. Hmm, probably a case of awkward NULL propagation.
2) Makes sense, I guess. EDIT: Done.Last modified on 2020-11-25 at 15:08
#14 by rampaa
2020-11-25 at 16:14
< report >
Nice! On that note, wouldn't merging "Sex" and "Sex (spoiler)" filters make sense? I think a spoiler toggle like we have for "Tags" filter would suffice and that way "Add filter" UI for characters would be less cluttered.


You must be logged in to reply to this thread.