Lucene - Core
  1. Lucene - Core
  2. LUCENE-5401

Field.StringTokenStream#end() does not call super.end()

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 4.6
    • Fix Version/s: 4.6.1, 4.7, 6.0
    • Component/s: core/other
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      Field.StringTokenStream#end() currently does not call super.end(). This prevents resetting the PositionIncrementAttribute to 0 in end(), which can lead to wrong positions in the index under certain conditions.

      I added a test to TestDocument which indexes two Fields with the same name, String values, indexed=true, tokenized=false and IndexOptions.DOCS_AND_FREQS_AND_POSITIONS. Without the fix the test fails. The first token gets the correct position 0, but the second token gets position 2 instead of 1. The reason is that in DocInverterPerField line 176 (which is just after the call to end()) we increment the position a second time, because end() didn't reset the increment to 0.

      All tests pass with the fix.

      1. lucene-5401.patch
        5 kB
        Michael Busch

        Activity

        Hide
        Robert Muir added a comment -

        +1, nice catch!

        Show
        Robert Muir added a comment - +1, nice catch!
        Hide
        Simon Willnauer added a comment -

        +1 can you backport this to 4.6.1 I think mark wants to take a RC1 build tonight! welcome back buschmi

        Show
        Simon Willnauer added a comment - +1 can you backport this to 4.6.1 I think mark wants to take a RC1 build tonight! welcome back buschmi
        Hide
        Mark Miller added a comment -

        +1, let's pop this in for 4.6.1.

        Show
        Mark Miller added a comment - +1, let's pop this in for 4.6.1.
        Hide
        ASF subversion and git services added a comment -

        Commit 1558876 from Michael Busch in branch 'dev/trunk'
        [ https://svn.apache.org/r1558876 ]

        LUCENE-5401: Field.StringTokenStream#end() calls super.end() now.

        Show
        ASF subversion and git services added a comment - Commit 1558876 from Michael Busch in branch 'dev/trunk' [ https://svn.apache.org/r1558876 ] LUCENE-5401 : Field.StringTokenStream#end() calls super.end() now.
        Hide
        ASF subversion and git services added a comment -

        Commit 1558877 from Michael Busch in branch 'dev/branches/lucene_solr_4_6'
        [ https://svn.apache.org/r1558877 ]

        LUCENE-5401: Field.StringTokenStream#end() calls super.end() now.

        Show
        ASF subversion and git services added a comment - Commit 1558877 from Michael Busch in branch 'dev/branches/lucene_solr_4_6' [ https://svn.apache.org/r1558877 ] LUCENE-5401 : Field.StringTokenStream#end() calls super.end() now.
        Hide
        ASF subversion and git services added a comment -

        Commit 1558878 from Michael Busch in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1558878 ]

        LUCENE-5401: Field.StringTokenStream#end() calls super.end() now.

        Show
        ASF subversion and git services added a comment - Commit 1558878 from Michael Busch in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1558878 ] LUCENE-5401 : Field.StringTokenStream#end() calls super.end() now.
        Hide
        Michael Busch added a comment -

        Thanks, guys! I backported to 4.6.1 and just committed. (feels good after a looong time )

        Show
        Michael Busch added a comment - Thanks, guys! I backported to 4.6.1 and just committed. (feels good after a looong time )

          People

          • Assignee:
            Michael Busch
            Reporter:
            Michael Busch
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development