[LUCENE-3113] fix analyzer bugs found by MockTokenizer

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.2, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Unassigned
• Reporter: Robert Muir
• Created: 17/May/11 14:15
• Updated: 28/Aug/22 12:47
• Resolved: 18/May/11 13:00

Description

In LUCENE-3064, we beefed up MockTokenizer with assertions, and I've switched over the analysis tests to use MockTokenizer for better coverage.

However, this found a few bugs (one of which is LUCENE-3106):

• incrementToken() after it returns false in CommonGramsQueryFilter, HyphenatedWordsFilter, ShingleFilter, SynonymFilter
• missing end() implementation for PrefixAwareTokenFilter
• double reset() in QueryAutoStopWordAnalyzer and ReusableAnalyzerBase
• missing correctOffset()s in MockTokenizer itself.

I think it would be nice to just fix all the bugs on one issue... I've fixed everything except Shingle and Synonym

Issue Links

• incorporates: LUCENE-3106 commongrams filter calls incrementToken() after it returns false