[LUCENE-2939] Highlighter should try and use maxDocCharsToAnalyze in WeightedSpanTermExtractor when adding a new field to MemoryIndex as well as when using CachingTokenStream

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.2, 4.0-ALPHA
• Component/s: modules/highlighter
• Assignee: Mark Miller
• Reporter: Mark Miller
• Created: 27/Feb/11 01:14
• Updated: 28/Aug/22 12:41
• Resolved: 20/Apr/11 14:44

Description

huge documents can be drastically slower than need be because the entire field is added to the memory index
this cost can be greatly reduced in many cases if we try and respect maxDocCharsToAnalyze

things can be improved even further by respecting this setting with CachingTokenStream

Issue Links

• is part of: SOLR-2390 Performance of usePhraseHighlighter is terrible on very large Documents, regardless of hl.maxDocCharsToAnalyze