[LUCENE-8348] Remove [Edge]NgramTokenizer min/max defaults consistent with Filter - ASF JIRA

XML

Word

Printable

JSON

Type: Task
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Component/s: modules/analysis
Labels:
None
Environment:

Hide

~~LUCENE-7960~~ fixed a good deal of trappiness here for the tokenfilters, there aren't ridiculous default min/max values such as 1,2.

Also javadocs are enhanced to present a nice warning about using large ranges: it seems to surprise people that min=small, max=huge eats up a ton of resources, but its really like creating (huge-small) separate n-gram indexes, so of course its expensive.

Finally it keeps it easy to do the typical, more efficient fixed ngram case, vs forcing someone to do min=X,max=X range which is unintuitive.

We should improve the tokenizers in the same way.

Show
LUCENE-7960 fixed a good deal of trappiness here for the tokenfilters, there aren't ridiculous default min/max values such as 1,2. Also javadocs are enhanced to present a nice warning about using large ranges: it seems to surprise people that min=small, max=huge eats up a ton of resources, but its really like creating (huge-small) separate n-gram indexes, so of course its expensive. Finally it keeps it easy to do the typical, more efficient fixed ngram case, vs forcing someone to do min=X,max=X range which is unintuitive. We should improve the tokenizers in the same way.