[SOLR-822] CharFilter - normalize characters before tokenizer

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.3
• Fix Version/s: 1.4
• Component/s: Schema and Analysis
• Assignee: Koji Sekiguchi
• Reporter: Koji Sekiguchi
• Created: 22/Oct/08 10:18
• Updated: 10/Jan/14 14:26
• Resolved: 19/Mar/09 11:52

Description

A new plugin which can be placed in front of <tokenizer/>.

<fieldType name="textCharNorm" class="solr.TextField" positionIncrementGap="100" >
  <analyzer>
    <charFilter class="solr.MappingCharFilterFactory" mapping="mapping_ja.txt" />
    <tokenizer class="solr.MappingCJKTokenizerFactory"/>
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>

<charFilter/> can be multiple (chained). I'll post a JPEG file to show character normalization sample soon.

MOTIVATION:
In Japan, there are two types of tokenizers – N-gram (CJKTokenizer) and Morphological Analyzer.
When we use morphological analyzer, because the analyzer uses Japanese dictionary to detect terms,
we need to normalize characters before tokenization.

I'll post a patch soon, too.