Details
-
Task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
LUCENE-7960fixed a good deal of trappiness here for the tokenfilters, there aren't ridiculous default min/max values such as 1,2.Also javadocs are enhanced to present a nice warning about using large ranges: it seems to surprise people that min=small, max=huge eats up a ton of resources, but its really like creating (huge-small) separate n-gram indexes, so of course its expensive.
Finally it keeps it easy to do the typical, more efficient fixed ngram case, vs forcing someone to do min=X,max=X range which is unintuitive.
We should improve the tokenizers in the same way.
LUCENE-7960 fixed a good deal of trappiness here for the tokenfilters, there aren't ridiculous default min/max values such as 1,2. Also javadocs are enhanced to present a nice warning about using large ranges: it seems to surprise people that min=small, max=huge eats up a ton of resources, but its really like creating (huge-small) separate n-gram indexes, so of course its expensive. Finally it keeps it easy to do the typical, more efficient fixed ngram case, vs forcing someone to do min=X,max=X range which is unintuitive. We should improve the tokenizers in the same way.
-
New