[STANBOL-849] Implement Lucene Tokenizer based LabelTokenizer

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: enhancement-engines-0.10.0
• Component/s: Enhancement Engines
• Assignee: Rupert Westenthaler
• Reporter: Rupert Westenthaler
• Created: 21/Dec/12 08:03
• Updated: 12/Apr/13 08:37
• Resolved: 22/Dec/12 06:06

Description

Lucene supports Tokenizers for a lot of languages. While the OpenNLP or Whitespace character based Tokenizers are fine for most of the languages this allows users to use special one (e.g. for Chinese the smartcn analyzer package)