[LUCENE-7526] Improvements to UnifiedHighlighter OffsetStrategies

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 6.4
• Component/s: modules/highlighter
• Lucene Fields: New
• Assignee: David Smiley
• Reporter: Timothy M. Rodriguez
• Created: 27/Oct/16 19:28
• Updated: 28/Aug/22 15:05
• Resolved: 15/Nov/16 21:21

Description

This ticket improves several of the UnifiedHighlighter FieldOffsetStrategies by reducing reliance on creating or re-creating TokenStreams.

The primary changes are as follows:

• AnalysisOffsetStrategy - split into two offset strategies
  • MemoryIndexOffsetStrategy - the primary analysis mode that utilizes a MemoryIndex for producing Offsets
  • TokenStreamOffsetStrategy - an offset strategy that avoids creating a MemoryIndex. Can only be used if the query distills down to terms and automata.

• TokenStream removal
  • MemoryIndexOffsetStrategy - previously a TokenStream was created to fill the memory index and then once consumed a new one was generated by uninverting the MemoryIndex back into a TokenStream if there were automata (wildcard/mtq queries) involved. Now this is avoided, which should save memory and avoid a second pass over the data.
  • TermVectorOffsetStrategy - this was refactored in a similar way to avoid generating a TokenStream if automata are involved.
  • PostingsWithTermVectorsOffsetStrategy - similar refactoring

• CompositePostingsEnum - aggregates several underlying PostingsEnums for wildcard/mtq queries. This should improve relevancy by providing unified metrics for a wildcard across all it's term matches

• Added a HighlightFlag for enabling the newly separated TokenStreamOffsetStrategy since it can adversely affect passage relevancy

Issue Links

• links to: GitHub Pull Request #105