Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7526

Improvements to UnifiedHighlighter OffsetStrategies


    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.4
    • Component/s: modules/highlighter
    • Labels:
    • Lucene Fields:


      This ticket improves several of the UnifiedHighlighter FieldOffsetStrategies by reducing reliance on creating or re-creating TokenStreams.

      The primary changes are as follows:

      • AnalysisOffsetStrategy - split into two offset strategies
        • MemoryIndexOffsetStrategy - the primary analysis mode that utilizes a MemoryIndex for producing Offsets
        • TokenStreamOffsetStrategy - an offset strategy that avoids creating a MemoryIndex. Can only be used if the query distills down to terms and automata.
      • TokenStream removal
        • MemoryIndexOffsetStrategy - previously a TokenStream was created to fill the memory index and then once consumed a new one was generated by uninverting the MemoryIndex back into a TokenStream if there were automata (wildcard/mtq queries) involved. Now this is avoided, which should save memory and avoid a second pass over the data.
        • TermVectorOffsetStrategy - this was refactored in a similar way to avoid generating a TokenStream if automata are involved.
        • PostingsWithTermVectorsOffsetStrategy - similar refactoring
      • CompositePostingsEnum - aggregates several underlying PostingsEnums for wildcard/mtq queries. This should improve relevancy by providing unified metrics for a wildcard across all it's term matches
      • Added a HighlightFlag for enabling the newly separated TokenStreamOffsetStrategy since it can adversely affect passage relevancy


          Issue Links



              • Assignee:
                dsmiley David Smiley
                Timothy055 Timothy M. Rodriguez
              • Votes:
                0 Vote for this issue
                4 Start watching this issue


                • Created: