[SOLR-1710] convert worddelimiterfilter to new tokenstream API

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1
• Component/s: Schema and Analysis
• Assignee: Mark Miller
• Reporter: Robert Muir
• Created: 08/Jan/10 13:08
• Updated: 30/Mar/11 15:45
• Resolved: 27/Mar/10 23:23

Description

This one was a doozy, attached is a patch to convert it to the new tokenstream API.

Some of the logic was split into WordDelimiterIterator (exposes a BreakIterator-like api for iterating subwords)
the filter is much more efficient now, no cloning.

before applying the patch, copy the existing WordDelimiterFilter to OriginalWordDelimiterFilter
the patch includes a testcase (TestWordDelimiterBWComp) which generates random strings from various subword combinations.
For each random string, it compares output against the existing WordDelimiterFilter for all 512 combinations of boolean parameters.

NOTE: due to bugs found (SOLR-1706), this currently only tests 256 of these combinations. The bugs discovered in SOLR-1706 are fixed here.

Issue Links

• incorporates: SOLR-1706 wrong tokens output from WordDelimiterFilter depending upon options
• is part of: SOLR-1657 convert the rest of solr to use the new tokenstream API