Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.2
    • Fix Version/s: 6.3
    • Component/s: modules/highlighter
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      The UnifiedHighlighter is an evolution of the PostingsHighlighter that is able to highlight using offsets in either postings, term vectors, or from analysis (a TokenStream). Lucene’s existing highlighters are mostly demarcated along offset source lines, whereas here it is unified – hence this proposed name. In this highlighter, the offset source strategy is separated from the core highlighting functionalty. The UnifiedHighlighter further improves on the PostingsHighlighter’s design by supporting accurate phrase highlighting using an approach similar to the standard highlighter’s WeightedSpanTermExtractor. The next major improvement is a hybrid offset source strategythat utilizes postings and “light” term vectors (i.e. just the terms) for highlighting multi-term queries (wildcards) without resorting to analysis. Phrase highlighting and wildcard highlighting can both be disabled if you’d rather highlight a little faster albeit not as accurately reflecting the query.
      We’ve benchmarked an earlier version of this highlighter comparing it to the other highlighters and the results were exciting! It’s tempting to share those results but it’s definitely due for another benchmark, so we’ll work on that. Performance was the main motivator for creating the UnifiedHighlighter, as the standard Highlighter (the only one meeting Bloomberg Law’s accuracy requirements) wasn’t fast enough, even with term vectors along with several improvements we contributed back, and even after we forked it to highlight in multiple threads.

        Attachments

        1. LUCENE_7438_UH_benchmark.patch
          58 kB
          David Smiley
        2. LUCENE_7438_UH_benchmark.patch
          28 kB
          David Smiley
        3. LUCENE_7438_UH_small_changes.patch
          9 kB
          David Smiley
        4. LUCENE-7438.patch
          393 kB
          David Smiley

          Issue Links

            Activity

              People

              • Assignee:
                dsmiley David Smiley
                Reporter:
                Timothy055 Timothy M. Rodriguez
              • Votes:
                8 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: