Lucene - Core
  1. Lucene - Core
  2. LUCENE-6382

Don't allow position = Integer.MAX_VALUE going forward

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.2, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Spinoff from LUCENE-6308, where Integer.MAX_VALUE position is now used as a sentinel during position iteration to indicate that there are no more positions.

      Where IW now detects int overflow of position, it should now also detect == Integer.MAX_VALUE.

      And CI should note corruption if a segment's version is >= 5.2 and has Integer.MAX_VALUE position.

      1. LUCENE-6382.patch
        23 kB
        Michael McCandless
      2. LUCENE-6382.patch
        19 kB
        Michael McCandless
      3. LUCENE-6382.patch
        15 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Patch.

        Adds IndexWriter.MAX_POSITION (= Integer.MAX_VALUE - 1), a test case confirming you can index that position but not +1 higher, and a change to CheckIndex to detect any position > MAX_POSITION as corruption if the segment version is >= 5.2.0.

        Show
        Michael McCandless added a comment - Patch. Adds IndexWriter.MAX_POSITION (= Integer.MAX_VALUE - 1), a test case confirming you can index that position but not +1 higher, and a change to CheckIndex to detect any position > MAX_POSITION as corruption if the segment version is >= 5.2.0.
        Hide
        Michael McCandless added a comment -

        Another iteration: I dropped the limit to Integer.MAX_VALUE - 128 (like MAX_DOC) in case this value is every an array length. I also fixed default PF to detect out-of-bounds positions on write, cleaned up the tests, and fixed CI to always note corruption in this case regardless of version. A couple nocommits tho...

        Show
        Michael McCandless added a comment - Another iteration: I dropped the limit to Integer.MAX_VALUE - 128 (like MAX_DOC) in case this value is every an array length. I also fixed default PF to detect out-of-bounds positions on write, cleaned up the tests, and fixed CI to always note corruption in this case regardless of version. A couple nocommits tho...
        Hide
        Michael McCandless added a comment -

        Another iteration, fixing the nocommits. I think it's ready.

        I added a test with a static (zip file) index containing a too-large position, and confirmed that both CheckIndex and IndexWriter (on merge) detect it.

        Show
        Michael McCandless added a comment - Another iteration, fixing the nocommits. I think it's ready. I added a test with a static (zip file) index containing a too-large position, and confirmed that both CheckIndex and IndexWriter (on merge) detect it.
        Hide
        Robert Muir added a comment -

        +1

        Maybe the static index test+index should sit in backwards/ so that it can go away in a future version when those indexes cannot be read?

        Show
        Robert Muir added a comment - +1 Maybe the static index test+index should sit in backwards/ so that it can go away in a future version when those indexes cannot be read?
        Hide
        Michael McCandless added a comment -

        Good idea Robert Muir I'll move it there.

        Show
        Michael McCandless added a comment - Good idea Robert Muir I'll move it there.
        Hide
        ASF subversion and git services added a comment -

        Commit 1673508 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1673508 ]

        LUCENE-6382: enforce max allowed indexed position

        Show
        ASF subversion and git services added a comment - Commit 1673508 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1673508 ] LUCENE-6382 : enforce max allowed indexed position
        Hide
        Anshum Gupta added a comment -

        Bulk close for 5.2.0.

        Show
        Anshum Gupta added a comment - Bulk close for 5.2.0.

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development