[LUCENE-5096] WhitespaceTokenizer supports Java whitespace, should also support Unicode whitespace

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 4.3.1
• Component/s: modules/analysis

Environment

all

• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Jörg Prante
• Created: 07/Jul/13 23:38
• Updated: 28/Aug/22 13:49
• Resolved: 08/Dec/15 00:05

Description

The whitespace tokenizer supports only Java whitespace as defined in http://docs.oracle.com/javase/6/docs/api/java/lang/Character.html#isWhitespace(char)

A useful improvement would be to support also Unicode whitespace as defined in the Unicode property list http://www.unicode.org/Public/UCD/latest/ucd/PropList.txt

Issue Links

• is duplicated by: LUCENE-6874 WhitespaceTokenizer should tokenize on NBSP