[UIMA-1447] Tabulations are annotated as tokens after a space

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3S
• Fix Version/s: 2.3S
• Component/s: Sandbox-WhitespaceTokenizer

Environment

Unix (ubuntu 8.04), Eclipse Galileo 3.5

• Assignee: Marshall Schor
• Reporter: Jérôme Rocheteau
• Created: 21/Jul/09 15:04
• Updated: 11/Aug/09 21:09
• Resolved: 11/Aug/09 21:09

Description

This is a test-text for the Whitespace Tokenizer in the UIMA Sandbox.
It behaves as follows: i.e. a '\t' character after a space is
annotated as a token and its covered text is set to the empty string ""!
I suppose it shoudn't be the case, am I wrong?