[LUCENE-4993] BeiderMorseFilter inserts tokens with positionIncrement=0, but ignores all custom attributes except OffsetAttribute

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.3
• Fix Version/s: 4.3.1, 6.0
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Uwe Schindler
• Reporter: Uwe Schindler
• Created: 09/May/13 22:36
• Updated: 28/Aug/22 13:46
• Resolved: 10/May/13 08:02

Description

BeiderMorseFilter inserts sometimes additional phonetic tokens for the same source token. Currently it calls clearAttributes before doing this and sets the new token's term, positionIncrement=0 and the original offset.

This leads to problems if the TokenStream contains other attributes inserted before (like KeywordAttribute, FlagsAttribute,...). Those are all reverted to defaults for the inserted tokens.

The TokenFilter should remove the special case done for preserving offsets and instead to captureState() and restoreState().

Issue Links

• requires: LUCENE-4994 PatternKeywordMarkerFilter is final and has protected ctor and cannot be instantiated by non-Lucene code