Lucene - Core
  1. Lucene - Core
  2. LUCENE-4993

BeiderMorseFilter inserts tokens with positionIncrement=0, but ignores all custom attributes except OffsetAttribute

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.3
    • Fix Version/s: 4.3.1, Trunk
    • Component/s: modules/analysis
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      BeiderMorseFilter inserts sometimes additional phonetic tokens for the same source token. Currently it calls clearAttributes before doing this and sets the new token's term, positionIncrement=0 and the original offset.

      This leads to problems if the TokenStream contains other attributes inserted before (like KeywordAttribute, FlagsAttribute,...). Those are all reverted to defaults for the inserted tokens.

      The TokenFilter should remove the special case done for preserving offsets and instead to captureState() and restoreState().

      1. LUCENE-4993.patch
        4 kB
        Uwe Schindler

        Issue Links

          Activity

          Hide
          Shalin Shekhar Mangar added a comment -

          Bulk closing after 4.3.1 release

          Show
          Shalin Shekhar Mangar added a comment - Bulk closing after 4.3.1 release
          Hide
          Shalin Shekhar Mangar added a comment -

          Back ported to 4.3.1 r1483376

          Show
          Shalin Shekhar Mangar added a comment - Back ported to 4.3.1 r1483376
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] uschindler
          http://svn.apache.org/viewvc?view=revision&revision=1480912

          Merged revision(s) 1480911 from lucene/dev/trunk:
          LUCENE-4993: Fix BeiderMorseFilter to preserve custom attributes when inserting tokens with position increment 0.

          Show
          Commit Tag Bot added a comment - [branch_4x commit] uschindler http://svn.apache.org/viewvc?view=revision&revision=1480912 Merged revision(s) 1480911 from lucene/dev/trunk: LUCENE-4993 : Fix BeiderMorseFilter to preserve custom attributes when inserting tokens with position increment 0.
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] uschindler
          http://svn.apache.org/viewvc?view=revision&revision=1480911

          LUCENE-4993: Fix BeiderMorseFilter to preserve custom attributes when inserting tokens with position increment 0.

          Show
          Commit Tag Bot added a comment - [trunk commit] uschindler http://svn.apache.org/viewvc?view=revision&revision=1480911 LUCENE-4993 : Fix BeiderMorseFilter to preserve custom attributes when inserting tokens with position increment 0.
          Hide
          Uwe Schindler added a comment -

          Simple patch including test.

          This filter now inserts tokens in the same way like the other phonetic filters.

          Show
          Uwe Schindler added a comment - Simple patch including test. This filter now inserts tokens in the same way like the other phonetic filters.

            People

            • Assignee:
              Uwe Schindler
              Reporter:
              Uwe Schindler
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development