[SOLR-2059] Allow customizing how WordDelimiterFilter tokenizes text.

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.1, 4.0-ALPHA
• Component/s: Schema and Analysis
• Assignee: Unassigned
• Reporter: Robert Muir
• Created: 19/Aug/10 13:48
• Updated: 30/Mar/11 15:45
• Resolved: 28/Aug/10 22:05

Description

By default, WordDelimiterFilter assigns 'types' to each character (computed from Unicode Properties).
Based on these types and the options provided, it splits and concatenates text.

In some circumstances, you might need to tweak the behavior of how this works.
It seems the filter already had this in mind, since you can pass in a custom byte[] type table.
But its not exposed in the factory.

I think you should be able to customize the defaults with a configuration file:

# A customized type mapping for WordDelimiterFilterFactory
# the allowable types are: LOWER, UPPER, ALPHA, DIGIT, ALPHANUM, SUBWORD_DELIM
# 
# the default for any character without a mapping is always computed from 
# Unicode character properties

# Map the $, %, '.', and ',' characters to DIGIT 
# This might be useful for financial data.
$ => DIGIT
% => DIGIT
. => DIGIT
\u002C => DIGIT