[LUCENE-5447] StandardTokenizer should break at consecutive chars matching Word_Break = MidLetter, MidNum and/or MidNumLet

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.6.1
• Fix Version/s: 4.7, 6.0
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Steve Rowe
• Reporter: Steve Rowe
• Created: 14/Feb/14 20:55
• Updated: 09/May/16 18:33
• Resolved: 19/Feb/14 17:59

Description

StandardTokenizer should split all of the following sequences into two tokens each, but they are all instead kept intact and output as single tokens:

"A::B"           (':' is in \p{Word_Break = MidLetter})
"1..2", "A..B"   ('.' is in \p{Word_Break = MidNumLet})
"A.:B"
"A:.B"
"1,,2"           (',' is in \p{Word_Break = MidNum})
"1,.2"
"1.,2"

Unfortunately, the word break test data released with Unicode, e.g. for Unicode 6.3 http://www.unicode.org/Public/6.3.0/ucd/auxiliary/WordBreakTest.txt, and incorporated into a versioned Lucene test, e.g. WordBreakTestUnicode_6_3_0, doesn't cover these cases.