[STANBOL-855] Add basic language support for Chinese

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: commons-0.11.0
• Component/s: Commons
• Assignee: Rupert Westenthaler
• Reporter: Rupert Westenthaler
• Created: 22/Dec/12 06:16
• Updated: 17/Jul/13 15:20
• Resolved: 24/Dec/12 13:02

Description

Basic language support includes:

• An Chinese tokenizer that creates Tokens for the AnalyzedText content part
• LabelTokenizer for the EntityLinking engine that supports Chinese text
• add support for Chinese specific Solr fieldType to the Stanbol Entityhub so that Chinese text is correctly indexed

This will bring basic EntityLinking support for Chinese texts to Apache Stanbol. This means that parsed Text is correctly tokenized and tokens can be matched with controlled vocabularies.

Issue Links

• relates to: STANBOL-875 Add support for Paoding (Chinese)

Sub-Tasks

• Add optional support for Chinese via the Solr/lucene smartcn module: Closed
• Add Lucene LabelTokenizer configuration for Chinese (zh) based on the Lucene smartcn analyzer: Closed
• Provide SolrCore configuration for the Entityhub SolrYard with smartcn fieldType configuration for Chinese (zh): Closed
• Add AnalyzedText Tokenizer based on the Smart Chinese Analyzer: Closed