Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-957

Lucene RAM Directory doesn't work for Index Size > 8 GB

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: core/store
    • Labels:
      None

      Description

      from user list - http://www.gossamer-threads.com/lists/lucene/java-user/50982

      Problem seems to be casting issues in RAMInputStream.

      Line 90:
      bufferStart = BUFFER_SIZE * currentBufferIndex;
      both rhs are ints while lhs is long.
      so a very large product would first overflow MAX_INT, become negative, and only then (auto) casted to long, but this is too late.

      Line 91:
      bufferLength = (int) (length - bufferStart);
      both rhs are longs while lhs is int.
      so the (int) cast result may turn negative and the logic that follows would be wrong.

      1. lucene-957.patch
        8 kB
        Doron Cohen
      2. lucene-957.patch
        1 kB
        Doron Cohen

        Activity

        Hide
        doronc Doron Cohen added a comment -

        Patch fixing int / long cast issues in RAMInputStream.

        All tests pass but non of those actually demonstrates the bug being fixed. I don't currently have a large enough index to test this.

        I will wait for a manual test by Murali (who reported this issue), and for at least one patch review.

        Show
        doronc Doron Cohen added a comment - Patch fixing int / long cast issues in RAMInputStream. All tests pass but non of those actually demonstrates the bug being fixed. I don't currently have a large enough index to test this. I will wait for a manual test by Murali (who reported this issue), and for at least one patch review.
        Hide
        doronc Doron Cohen added a comment -

        Previous patch apparently did not fix the bug - a casting problem in RAMOutputStream had to be fixed.
        Updated patch adds a test imitating ramFile larger than maxint.
        For this had to make the allocation of a new byte array in RAMFile overridable.
        The new test fails before fixing RAMOutputStream (affecting RAMDirectory constructor from FS). However the issues in RAMInputStream in fact do not cause failures, yet they should be fixed.

        With a test in place I now feel confident in this fix - will commit it in a day or two if there are no reservations.

        Show
        doronc Doron Cohen added a comment - Previous patch apparently did not fix the bug - a casting problem in RAMOutputStream had to be fixed. Updated patch adds a test imitating ramFile larger than maxint. For this had to make the allocation of a new byte array in RAMFile overridable. The new test fails before fixing RAMOutputStream (affecting RAMDirectory constructor from FS). However the issues in RAMInputStream in fact do not cause failures, yet they should be fixed. With a test in place I now feel confident in this fix - will commit it in a day or two if there are no reservations.
        Hide
        doronc Doron Cohen added a comment -

        committed.

        Show
        doronc Doron Cohen added a comment - committed.
        Hide
        muraalee Murali Varadarajan added a comment -

        Doron Cohon,

        Thanks a tonne. I tried your first patch and it didn't work. Today i saw your message and checked out the source from head and built it.
        It worked !!!

        Show
        muraalee Murali Varadarajan added a comment - Doron Cohon, Thanks a tonne. I tried your first patch and it didn't work. Today i saw your message and checked out the source from head and built it. It worked !!!

          People

          • Assignee:
            doronc Doron Cohen
            Reporter:
            doronc Doron Cohen
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development