[LUCENE-5161] review FSDirectory chunking defaults and test the chunking

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.5, 6.0
• Lucene Fields: New
• Assignee: Robert Muir
• Reporter: Robert Muir
• Created: 08/Aug/13 20:13
• Updated: 28/Aug/22 13:51
• Resolved: 10/Aug/13 14:30

Description

Today there is a loop in SimpleFS/NIOFS:

try {
          do {
            final int readLength;
            if (total + chunkSize > len) {
              readLength = len - total;
            } else {
              // LUCENE-1566 - work around JVM Bug by breaking very large reads into chunks
              readLength = chunkSize;
            }
            final int i = file.read(b, offset + total, readLength);
            total += i;
          } while (total < len);
        } catch (OutOfMemoryError e) {

I bet if you look at the clover report its untested, because its fixed at 100MB for 32-bit users and 2GB for 64-bit users (are these defaults even good?!).

Also if you call the setter on a 64-bit machine to change the size, it just totally ignores it. We should remove that, the setter should always work.

And we should set it to small values in tests so this loop is actually executed.

Issue Links

• is related to: LUCENE-5160 NIOFSDirectory, SimpleFSDirectory (others?) don't properly handle valid file and FileChannel read conditions
• is required by: LUCENE-5164 Remove the OOM catching in SimpleFSDirectory and NIOFSDirectory