Lucene RAM Directory doesn't work for Index Size > 8 GB

• Key: LUCENE-957
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Doron Cohen
• Reporter: Doron Cohen
• Component/s: Store

Description

from user list - http://www.gossamer-threads.com/lists/lucene/java-user/50982

Problem seems to be casting issues in RAMInputStream.

Line 90:
bufferStart = BUFFER_SIZE * currentBufferIndex;
both rhs are ints while lhs is long.
so a very large product would first overflow MAX_INT, become negative, and only then (auto) casted to long, but this is too late.

Line 91:
bufferLength = (int) (length - bufferStart);
both rhs are longs while lhs is int.
so the (int) cast result may turn negative and the logic that follows would be wrong.

Comments

[Doron Cohen]

Patch fixing int / long cast issues in RAMInputStream.

All tests pass but non of those actually demonstrates the bug being fixed. I don't currently have a large enough index to test this.

I will wait for a manual test by Murali (who reported this issue), and for at least one patch review.

[Doron Cohen]

Previous patch apparently did not fix the bug - a casting problem in RAMOutputStream had to be fixed.
Updated patch adds a test imitating ramFile larger than maxint.
For this had to make the allocation of a new byte array in RAMFile overridable.
The new test fails before fixing RAMOutputStream (affecting RAMDirectory constructor from FS). However the issues in RAMInputStream in fact do not cause failures, yet they should be fixed.

With a test in place I now feel confident in this fix - will commit it in a day or two if there are no reservations.

[Doron Cohen]

committed.

[Murali Varadarajan]

Doron Cohon,

Thanks a tonne. I tried your first patch and it didn't work. Today i saw your message and checked out the source from head and built it.
It worked !!!