[LUCENE-5278] MockTokenizer throws away the character right after a token even if it is a valid start to a new token

• Type: Bug
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Fix Version/s: 4.6, 6.0
• Lucene Fields: New
• Assignee: Robert Muir
• Reporter: Nik Everett
• Created: 11/Oct/13 19:25
• Updated: 28/Aug/22 13:55
• Resolved: 12/Oct/13 04:31

Description

MockTokenizer throws away the character right after a token even if it is a valid start to a new token. You won't see this unless you build a tokenizer that can recognize every character like with new RegExp(".") or RegExp("...").

Changing this behaviour seems to break a number of tests.