[LUCENE-2219] improve BaseTokenStreamTestCase to test end()

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9, 3.0
• Fix Version/s: 2.9.2, 3.0.1, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Robert Muir
• Reporter: Robert Muir
• Created: 17/Jan/10 08:15
• Updated: 28/Aug/22 12:19
• Resolved: 17/Jan/10 21:45

Description

If offsetAtt/end() is not implemented correctly, then there can be problems with highlighting: see LUCENE-2207 for an example with CJKTokenizer.

In my opinion you currently have to write too much code to test this.

This patch does the following:

• adds optional Integer finalOffset (can be null for no checking) to assertTokenStreamContents
• in assertAnalyzesTo, automatically fill this with the String length()

In my opinion this is correct, for assertTokenStreamContents the behavior should be optional, it may not even have a Tokenizer. If you are using assertTokenStreamContents with a Tokenizer then simply provide the extra expected value to check it.

for assertAnalyzesTo then it is implied there is a tokenizer so it should be checked.

the tests pass for core but there are failures in contrib even besides CJKTokenizer (apply Koji's patch from LUCENE-2207, it is correct). Specifically ChineseTokenizer has a similar problem.

Issue Links

• incorporates: LUCENE-2207 CJKTokenizer generates tokens with incorrect offsets