[LUCENE-2874] Highlighting overlapping tokens outputs doubled words

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.4
• Fix Version/s: 2.9.5, 3.0.4, 3.1, 4.0-ALPHA
• Component/s: modules/highlighter
• Lucene Fields: New, Patch Available
• Assignee: Robert Muir
• Reporter: Pierre Gossé
• Created: 19/Jan/11 10:41
• Updated: 28/Aug/22 12:39
• Resolved: 19/Jan/11 12:37

Description

If for the text "the fox did not jump" we generate following tokens :
(the, 0, 0-3),(

{fox}

,0,0-7),(fox,1,4-7),(did,2,8-11),(not,3,12,15),(jump,4,16,18)

If TermVector for field is stored WITH_OFFSETS and not WITH_POSITIONS_OFFSETS, highlighing would output
"the<em>the fox</em> did not jump"

I join a patch with 2 additive JUnit tests and a fix of TokenSources class where token ordering by offset did'nt manage well overlapping tokens.