[LUCENE-6874] WhitespaceTokenizer should tokenize on NBSP

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 5.4, 6.0
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Uwe Schindler
• Reporter: David Smiley
• Created: 02/Nov/15 05:10
• Updated: 28/Aug/22 14:45
• Resolved: 14/Nov/15 19:33

Description

WhitespaceTokenizer uses Character.isWhitespace to decide what is whitespace. Here's a pertinent excerpt:

It is a Unicode space character (SPACE_SEPARATOR, LINE_SEPARATOR, or PARAGRAPH_SEPARATOR) but is not also a non-breaking space ('\u00A0', '\u2007', '\u202F')

Perhaps Character.isWhitespace should have been called isLineBreakableWhitespace?

I think WhitespaceTokenizer should tokenize on this. I am aware it's easy to work around but why leave this trap in by default?

Issue Links

• duplicates: LUCENE-5096 WhitespaceTokenizer supports Java whitespace, should also support Unicode whitespace
• relates to: LUCENE-6879 Allow to define custom CharTokenizer using Java 8 Lambdas/Method references