[LUCENE-4291] consider reducing jflex buffer sizes

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0-BETA, 6.0
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Robert Muir
• Created: 06/Aug/12 15:14
• Updated: 28/Aug/22 13:24
• Resolved: 06/Aug/12 17:37

Description

Spinoff from SOLR-3684.

Most lucene tokenizers have some buffer size, e.g. in CharTokenizer/ICUTokenizer its char[4096].

But the jflex tokenizers use char[16384] by default, which seems overkill. I'm not sure we really see any performance bonus by having such a huge buffer size as a default.

There is a jflex parameter to set this: I think we should consider reducing it.

In a configuration like solr, tokenizers are reused per-thread-per-field,
so these can easily stack up in RAM.

Additionally CharFilters are not reused so the configuration in e.g.
HtmlStripCharFilter might not be great since its per-document garbage.