[SOLR-532] WordDelimiterFilter ignores payloads

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.4
• Assignee: Grant Ingersoll
• Reporter: Tricia Jenkins
• Created: 03/Apr/08 16:04
• Updated: 10/Nov/09 15:51
• Resolved: 22/Oct/08 12:34

Description

When a WordDelimiterFilter ingests a token stream and creates a new token (newTok) it appears to copy most of the old token attributes, except the payload. I believe this is a bug. My solution is for the WordDelimiterFilter to use the Token clone() method to create a carbon copy and then modify the appropriate attributes (offsets and term text).

Issue Links

• is related to: LUCENE-1350 Filters which are "consumers" should not reset the payload or flags and should better reuse the token
• is related to: SOLR-380 There's no way to convert search results into page-level hits of a "structured document".