Index: . =================================================================== --- . (revision 1512948) +++ . (working copy) Property changes on: . ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk:r1512937 Index: dev-tools =================================================================== --- dev-tools (revision 1512948) +++ dev-tools (working copy) Property changes on: dev-tools ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/dev-tools:r1512937 Index: lucene =================================================================== --- lucene (revision 1512948) +++ lucene (working copy) Property changes on: lucene ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene:r1512937 Index: lucene/analysis =================================================================== --- lucene/analysis (revision 1512948) +++ lucene/analysis (working copy) Property changes on: lucene/analysis ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/analysis:r1512937 Index: lucene/analysis/icu/src/java/org/apache/lucene/collation/ICUCollationKeyFilterFactory.java =================================================================== --- lucene/analysis/icu/src/java/org/apache/lucene/collation/ICUCollationKeyFilterFactory.java (revision 1512948) +++ lucene/analysis/icu/src/java/org/apache/lucene/collation/ICUCollationKeyFilterFactory.java (working copy) Property changes on: lucene/analysis/icu/src/java/org/apache/lucene/collation/ICUCollationKeyFilterFactory.java ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/analysis/icu/src/java/org/apache/lucene/collation/ICUCollationKeyFilterFactory.java:r1512937 Index: lucene/backwards =================================================================== --- lucene/backwards (revision 1512948) +++ lucene/backwards (working copy) Property changes on: lucene/backwards ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/backwards:r1512937 Index: lucene/benchmark =================================================================== --- lucene/benchmark (revision 1512948) +++ lucene/benchmark (working copy) Property changes on: lucene/benchmark ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/benchmark:r1512937 Index: lucene/BUILD.txt =================================================================== --- lucene/BUILD.txt (revision 1512948) +++ lucene/BUILD.txt (working copy) Property changes on: lucene/BUILD.txt ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/BUILD.txt:r1512937 Index: lucene/build.xml =================================================================== --- lucene/build.xml (revision 1512948) +++ lucene/build.xml (working copy) Property changes on: lucene/build.xml ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/build.xml:r1512937 Index: lucene/CHANGES.txt =================================================================== --- lucene/CHANGES.txt (revision 1512948) +++ lucene/CHANGES.txt (working copy) @@ -63,14 +63,11 @@ seek/lookup which can cause sideeffects if done on a cached FST root arc. (Simon Willnauer) -* LUCENE-5160: Handle the case where reading from a file or FileChannel returns -1, which - could happen in rare cases where something happens to the file between the time we start the - read loop (where we check the length) and when we actually do the read. (gsingers, yonik, Robert Muir, Uwe Schindler) +* LUCENE-5160: Handle the case where reading from a file or FileChannel returns -1, + which could happen in rare cases where something happens to the file between the + time we start the read loop (where we check the length) and when we actually do + the read. (gsingers, yonik, Robert Muir, Uwe Schindler) -* LUCENE-5161: Fix default chunk sizes in FSDirectory.java to not be unnecessarily large, - and fix setReadChunkSize to always work regardless of whether the machine is 32bit - or 64bit. (Uwe Schindler, Robert Muir) - API Changes * LUCENE-5094: Add ramBytesUsed() to MultiDocValues.OrdinalMap. @@ -106,6 +103,11 @@ * LUCENE-5156: CompressingTermVectors TermsEnum no longer supports ord(). (Robert Muir) +* LUCENE-5161, LUCENE-5164: Fix default chunk sizes in FSDirectory to not be + unnecessarily large (now 8192 bytes); also use chunking when writing to index + files. FSDirectory#setReadChunkSize() is now deprecated and will be removed + in Lucene 5.0. (Uwe Schindler, Robert Muir, gsingers) + Optimizations * LUCENE-5088: Added TermFilter to filter docs by a specific term. Index: lucene/CHANGES.txt =================================================================== --- lucene/CHANGES.txt (revision 1512948) +++ lucene/CHANGES.txt (working copy) Property changes on: lucene/CHANGES.txt ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/CHANGES.txt:r1512937 Index: lucene/classification =================================================================== --- lucene/classification (revision 1512948) +++ lucene/classification (working copy) Property changes on: lucene/classification ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/classification:r1512937 Index: lucene/classification/build.xml =================================================================== --- lucene/classification/build.xml (revision 1512948) +++ lucene/classification/build.xml (working copy) Property changes on: lucene/classification/build.xml ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/classification/build.xml:r1512937 Index: lucene/classification/ivy.xml =================================================================== --- lucene/classification/ivy.xml (revision 1512948) +++ lucene/classification/ivy.xml (working copy) Property changes on: lucene/classification/ivy.xml ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/classification/ivy.xml:r1512937 Index: lucene/classification/src =================================================================== --- lucene/classification/src (revision 1512948) +++ lucene/classification/src (working copy) Property changes on: lucene/classification/src ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/classification/src:r1512937 Index: lucene/codecs =================================================================== --- lucene/codecs (revision 1512948) +++ lucene/codecs (working copy) Property changes on: lucene/codecs ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/codecs:r1512937 Index: lucene/common-build.xml =================================================================== --- lucene/common-build.xml (revision 1512948) +++ lucene/common-build.xml (working copy) Property changes on: lucene/common-build.xml ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/common-build.xml:r1512937 Index: lucene/core =================================================================== --- lucene/core (revision 1512948) +++ lucene/core (working copy) Property changes on: lucene/core ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/core:r1512937 Index: lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java =================================================================== --- lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java (revision 1512948) +++ lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java (working copy) @@ -23,7 +23,7 @@ /** Base implementation class for buffered {@link IndexInput}. */ public abstract class BufferedIndexInput extends IndexInput { - /** Default buffer size set to 1024*/ + /** Default buffer size set to {@value #BUFFER_SIZE}. */ public static final int BUFFER_SIZE = 1024; // The normal read buffer size defaults to 1024, but @@ -33,7 +33,7 @@ // BufferedIndexInputs created during merging. See // LUCENE-888 for details. /** - * A buffer size for merges set to 4096 + * A buffer size for merges set to {@value #MERGE_BUFFER_SIZE}. */ public static final int MERGE_BUFFER_SIZE = 4096; @@ -115,15 +115,14 @@ @Override public final void readBytes(byte[] b, int offset, int len, boolean useBuffer) throws IOException { - - if(len <= (bufferLength-bufferPosition)){ + int available = bufferLength - bufferPosition; + if(len <= available){ // the buffer contains enough data to satisfy this request if(len>0) // to allow b to be null if len is 0... System.arraycopy(buffer, bufferPosition, b, offset, len); bufferPosition+=len; } else { // the buffer does not have enough data. First serve all we've got. - int available = bufferLength - bufferPosition; if(available > 0){ System.arraycopy(buffer, bufferPosition, b, offset, available); offset += available; Index: lucene/core/src/java/org/apache/lucene/store/FSDirectory.java =================================================================== --- lucene/core/src/java/org/apache/lucene/store/FSDirectory.java (revision 1512948) +++ lucene/core/src/java/org/apache/lucene/store/FSDirectory.java (working copy) @@ -31,6 +31,7 @@ import org.apache.lucene.util.ThreadInterruptedException; import org.apache.lucene.util.Constants; +import org.apache.lucene.util.IOUtils; /** * Base class for Directory implementations that store index @@ -112,13 +113,16 @@ public abstract class FSDirectory extends Directory { /** - * Default read chunk size: 2*{@link BufferedIndexInput#MERGE_BUFFER_SIZE}. + * Default read chunk size: 8192 bytes (this is the size up to which the JDK + does not allocate additional arrays while reading/writing) + @deprecated This constant is no longer used since Lucene 4.5. */ - public static final int DEFAULT_READ_CHUNK_SIZE = BufferedIndexInput.MERGE_BUFFER_SIZE * 2; + @Deprecated + public static final int DEFAULT_READ_CHUNK_SIZE = 8192; protected final File directory; // The underlying filesystem directory protected final Set staleFiles = synchronizedSet(new HashSet()); // Files written, but not yet sync'ed - private int chunkSize = DEFAULT_READ_CHUNK_SIZE; // LUCENE-1566 + private int chunkSize = DEFAULT_READ_CHUNK_SIZE; // returns the canonical version of the directory, creating it if it doesn't exist. private static File getCanonicalPath(File file) throws IOException { @@ -348,24 +352,11 @@ } /** - * Sets the maximum number of bytes read at once from the - * underlying file during {@link IndexInput#readBytes}. - * The default value is {@link #DEFAULT_READ_CHUNK_SIZE}; - * - *

This was introduced due to Sun - * JVM Bug 6478546, which throws an incorrect - * OutOfMemoryError when attempting to read too many bytes - * at once. It only happens on 32bit JVMs with a large - * maximum heap size.

- * - *

Changes to this value will not impact any - * already-opened {@link IndexInput}s. You should call - * this before attempting to open an index on the - * directory.

+ * This setting has no effect anymore. + * @deprecated This is no longer used since Lucene 4.5. */ + @Deprecated public final void setReadChunkSize(int chunkSize) { - // LUCENE-1566 if (chunkSize <= 0) { throw new IllegalArgumentException("chunkSize must be positive"); } @@ -373,12 +364,11 @@ } /** - * The maximum number of bytes to read at once from the - * underlying file during {@link IndexInput#readBytes}. - * @see #setReadChunkSize + * This setting has no effect anymore. + * @deprecated This is no longer used since Lucene 4.5. */ + @Deprecated public final int getReadChunkSize() { - // LUCENE-1566 return chunkSize; } @@ -387,27 +377,23 @@ /** the underlying RandomAccessFile */ protected final RandomAccessFile file; boolean isClone = false; - /** maximum read length on a 32bit JVM to prevent incorrect OOM, see LUCENE-1566 */ - protected final int chunkSize; /** start offset: non-zero in the slice case */ protected final long off; /** end offset (start+length) */ protected final long end; /** Create a new FSIndexInput, reading the entire file from path */ - protected FSIndexInput(String resourceDesc, File path, IOContext context, int chunkSize) throws IOException { + protected FSIndexInput(String resourceDesc, File path, IOContext context) throws IOException { super(resourceDesc, context); this.file = new RandomAccessFile(path, "r"); - this.chunkSize = chunkSize; this.off = 0L; this.end = file.length(); } /** Create a new FSIndexInput, representing a slice of an existing open file */ - protected FSIndexInput(String resourceDesc, RandomAccessFile file, long off, long length, int bufferSize, int chunkSize) { + protected FSIndexInput(String resourceDesc, RandomAccessFile file, long off, long length, int bufferSize) { super(resourceDesc, bufferSize); this.file = file; - this.chunkSize = chunkSize; this.off = off; this.end = off + length; this.isClone = true; // well, we are sorta? @@ -445,23 +431,35 @@ * Writes output with {@link RandomAccessFile#write(byte[], int, int)} */ protected static class FSIndexOutput extends BufferedIndexOutput { + /** + * The maximum chunk size is 8192 bytes, because {@link RandomAccessFile} mallocs + * a native buffer outside of stack if the write buffer size is larger. + */ + private static final int CHUNK_SIZE = 8192; + private final FSDirectory parent; private final String name; private final RandomAccessFile file; private volatile boolean isOpen; // remember if the file is open, so that we don't try to close it more than once public FSIndexOutput(FSDirectory parent, String name) throws IOException { + super(CHUNK_SIZE); this.parent = parent; this.name = name; file = new RandomAccessFile(new File(parent.directory, name), "rw"); isOpen = true; } - /** output methods: */ @Override - public void flushBuffer(byte[] b, int offset, int size) throws IOException { + protected void flushBuffer(byte[] b, int offset, int size) throws IOException { assert isOpen; - file.write(b, offset, size); + while (size > 0) { + final int toWrite = Math.min(CHUNK_SIZE, size); + file.write(b, offset, toWrite); + offset += toWrite; + size -= toWrite; + } + assert size == 0; } @Override @@ -469,21 +467,14 @@ parent.onIndexOutputClosed(this); // only close the file if it has not been closed yet if (isOpen) { - boolean success = false; + IOException priorE = null; try { super.close(); - success = true; + } catch (IOException ioe) { + priorE = ioe; } finally { isOpen = false; - if (!success) { - try { - file.close(); - } catch (Throwable t) { - // Suppress so we don't mask original exception - } - } else { - file.close(); - } + IOUtils.closeWhileHandlingException(priorE, file); } } } Index: lucene/core/src/java/org/apache/lucene/store/NIOFSDirectory.java =================================================================== --- lucene/core/src/java/org/apache/lucene/store/NIOFSDirectory.java (revision 1512948) +++ lucene/core/src/java/org/apache/lucene/store/NIOFSDirectory.java (working copy) @@ -77,7 +77,7 @@ @Override public IndexInput openInput(String name, IOContext context) throws IOException { ensureOpen(); - return new NIOFSIndexInput(new File(getDirectory(), name), context, getReadChunkSize()); + return new NIOFSIndexInput(new File(getDirectory(), name), context); } @Override @@ -96,7 +96,7 @@ @Override public IndexInput openSlice(String sliceDescription, long offset, long length) { return new NIOFSIndexInput(sliceDescription, path, descriptor, descriptor.getChannel(), offset, - length, BufferedIndexInput.bufferSize(context), getReadChunkSize()); + length, BufferedIndexInput.bufferSize(context)); } @Override @@ -114,18 +114,22 @@ * Reads bytes with {@link FileChannel#read(ByteBuffer, long)} */ protected static class NIOFSIndexInput extends FSIndexInput { - + /** + * The maximum chunk size for reads of 16384 bytes. + */ + private static final int CHUNK_SIZE = 16384; + private ByteBuffer byteBuf; // wraps the buffer for NIO final FileChannel channel; - public NIOFSIndexInput(File path, IOContext context, int chunkSize) throws IOException { - super("NIOFSIndexInput(path=\"" + path + "\")", path, context, chunkSize); + public NIOFSIndexInput(File path, IOContext context) throws IOException { + super("NIOFSIndexInput(path=\"" + path + "\")", path, context); channel = file.getChannel(); } - public NIOFSIndexInput(String sliceDescription, File path, RandomAccessFile file, FileChannel fc, long off, long length, int bufferSize, int chunkSize) { - super("NIOFSIndexInput(" + sliceDescription + " in path=\"" + path + "\" slice=" + off + ":" + (off+length) + ")", file, off, length, bufferSize, chunkSize); + public NIOFSIndexInput(String sliceDescription, File path, RandomAccessFile file, FileChannel fc, long off, long length, int bufferSize) { + super("NIOFSIndexInput(" + sliceDescription + " in path=\"" + path + "\" slice=" + off + ":" + (off+length) + ")", file, off, length, bufferSize); channel = fc; isClone = true; } @@ -138,24 +142,18 @@ @Override protected void readInternal(byte[] b, int offset, int len) throws IOException { - final ByteBuffer bb; // Determine the ByteBuffer we should use - if (b == buffer && 0 == offset) { + if (b == buffer) { // Use our own pre-wrapped byteBuf: assert byteBuf != null; - byteBuf.clear(); - byteBuf.limit(len); bb = byteBuf; + byteBuf.clear().position(offset); } else { bb = ByteBuffer.wrap(b, offset, len); } - int readOffset = bb.position(); - int readLength = bb.limit() - readOffset; - assert readLength == len; - long pos = getFilePointer() + off; if (pos + len > end) { @@ -163,33 +161,20 @@ } try { + int readLength = len; while (readLength > 0) { - final int limit; - if (readLength > chunkSize) { - // LUCENE-1566 - work around JVM Bug by breaking - // very large reads into chunks - limit = readOffset + chunkSize; - } else { - limit = readOffset + readLength; + final int toRead = Math.min(CHUNK_SIZE, readLength); + bb.limit(bb.position() + toRead); + assert bb.remaining() == toRead; + final int i = channel.read(bb, pos); + if (i < 0) { // be defensive here, even though we checked before hand, something could have changed + throw new EOFException("read past EOF: " + this + " off: " + offset + " len: " + len + " pos: " + pos + " chunkLen: " + toRead + " end: " + end); } - bb.limit(limit); - int i = channel.read(bb, pos); - if (i < 0){//be defensive here, even though we checked before hand, something could have changed - throw new EOFException("read past EOF: " + this + " off: " + offset + " len: " + len + " pos: " + pos + " limit: " + limit + " end: " + end); - } + assert i > 0 : "FileChannel.read with non zero-length bb.remaining() must always read at least one byte (FileChannel is in blocking mode, see spec of ReadableByteChannel)"; pos += i; - readOffset += i; readLength -= i; } - } catch (OutOfMemoryError e) { - // propagate OOM up and add a hint for 32bit VM Users hitting the bug - // with a large chunk size in the fast path. - final OutOfMemoryError outOfMemoryError = new OutOfMemoryError( - "OutOfMemoryError likely caused by the Sun VM Bug described in " - + "https://issues.apache.org/jira/browse/LUCENE-1566; try calling FSDirectory.setReadChunkSize " - + "with a value smaller than the current chunk size (" + chunkSize + ")"); - outOfMemoryError.initCause(e); - throw outOfMemoryError; + assert readLength == 0; } catch (IOException ioe) { throw new IOException(ioe.getMessage() + ": " + this, ioe); } Index: lucene/core/src/java/org/apache/lucene/store/SimpleFSDirectory.java =================================================================== --- lucene/core/src/java/org/apache/lucene/store/SimpleFSDirectory.java (revision 1512948) +++ lucene/core/src/java/org/apache/lucene/store/SimpleFSDirectory.java (working copy) @@ -55,7 +55,7 @@ public IndexInput openInput(String name, IOContext context) throws IOException { ensureOpen(); final File path = new File(directory, name); - return new SimpleFSIndexInput("SimpleFSIndexInput(path=\"" + path.getPath() + "\")", path, context, getReadChunkSize()); + return new SimpleFSIndexInput("SimpleFSIndexInput(path=\"" + path.getPath() + "\")", path, context); } @Override @@ -74,7 +74,7 @@ @Override public IndexInput openSlice(String sliceDescription, long offset, long length) { return new SimpleFSIndexInput("SimpleFSIndexInput(" + sliceDescription + " in path=\"" + file.getPath() + "\" slice=" + offset + ":" + (offset+length) + ")", descriptor, offset, - length, BufferedIndexInput.bufferSize(context), getReadChunkSize()); + length, BufferedIndexInput.bufferSize(context)); } @Override @@ -93,13 +93,18 @@ * {@link RandomAccessFile#read(byte[], int, int)}. */ protected static class SimpleFSIndexInput extends FSIndexInput { + /** + * The maximum chunk size is 8192 bytes, because {@link RandomAccessFile} mallocs + * a native buffer outside of stack if the read buffer size is larger. + */ + private static final int CHUNK_SIZE = 8192; - public SimpleFSIndexInput(String resourceDesc, File path, IOContext context, int chunkSize) throws IOException { - super(resourceDesc, path, context, chunkSize); + public SimpleFSIndexInput(String resourceDesc, File path, IOContext context) throws IOException { + super(resourceDesc, path, context); } - public SimpleFSIndexInput(String resourceDesc, RandomAccessFile file, long off, long length, int bufferSize, int chunkSize) { - super(resourceDesc, file, off, length, bufferSize, chunkSize); + public SimpleFSIndexInput(String resourceDesc, RandomAccessFile file, long off, long length, int bufferSize) { + super(resourceDesc, file, off, length, bufferSize); } /** IndexInput methods */ @@ -116,29 +121,16 @@ } try { - do { - final int readLength; - if (total + chunkSize > len) { - readLength = len - total; - } else { - // LUCENE-1566 - work around JVM Bug by breaking very large reads into chunks - readLength = chunkSize; + while (total < len) { + final int toRead = Math.min(CHUNK_SIZE, len - total); + final int i = file.read(b, offset + total, toRead); + if (i < 0) { // be defensive here, even though we checked before hand, something could have changed + throw new EOFException("read past EOF: " + this + " off: " + offset + " len: " + len + " total: " + total + " chunkLen: " + toRead + " end: " + end); } - final int i = file.read(b, offset + total, readLength); - if (i < 0){//be defensive here, even though we checked before hand, something could have changed - throw new EOFException("read past EOF: " + this + " off: " + offset + " len: " + len + " total: " + total + " readLen: " + readLength + " end: " + end); - } + assert i > 0 : "RandomAccessFile.read with non zero-length toRead must always read at least one byte"; total += i; - } while (total < len); - } catch (OutOfMemoryError e) { - // propagate OOM up and add a hint for 32bit VM Users hitting the bug - // with a large chunk size in the fast path. - final OutOfMemoryError outOfMemoryError = new OutOfMemoryError( - "OutOfMemoryError likely caused by the Sun VM Bug described in " - + "https://issues.apache.org/jira/browse/LUCENE-1566; try calling FSDirectory.setReadChunkSize " - + "with a value smaller than the current chunk size (" + chunkSize + ")"); - outOfMemoryError.initCause(e); - throw outOfMemoryError; + } + assert total == len; } catch (IOException ioe) { throw new IOException(ioe.getMessage() + ": " + this, ioe); } Index: lucene/core/src/test/org/apache/lucene/index/index.40.cfs.zip =================================================================== --- lucene/core/src/test/org/apache/lucene/index/index.40.cfs.zip (revision 1512948) +++ lucene/core/src/test/org/apache/lucene/index/index.40.cfs.zip (working copy) Property changes on: lucene/core/src/test/org/apache/lucene/index/index.40.cfs.zip ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/core/src/test/org/apache/lucene/index/index.40.cfs.zip:r1512937 Index: lucene/core/src/test/org/apache/lucene/index/index.40.nocfs.zip =================================================================== --- lucene/core/src/test/org/apache/lucene/index/index.40.nocfs.zip (revision 1512948) +++ lucene/core/src/test/org/apache/lucene/index/index.40.nocfs.zip (working copy) Property changes on: lucene/core/src/test/org/apache/lucene/index/index.40.nocfs.zip ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/core/src/test/org/apache/lucene/index/index.40.nocfs.zip:r1512937 Index: lucene/core/src/test/org/apache/lucene/index/index.40.optimized.cfs.zip =================================================================== --- lucene/core/src/test/org/apache/lucene/index/index.40.optimized.cfs.zip (revision 1512948) +++ lucene/core/src/test/org/apache/lucene/index/index.40.optimized.cfs.zip (working copy) Property changes on: lucene/core/src/test/org/apache/lucene/index/index.40.optimized.cfs.zip ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/core/src/test/org/apache/lucene/index/index.40.optimized.cfs.zip:r1512937 Index: lucene/core/src/test/org/apache/lucene/index/index.40.optimized.nocfs.zip =================================================================== --- lucene/core/src/test/org/apache/lucene/index/index.40.optimized.nocfs.zip (revision 1512948) +++ lucene/core/src/test/org/apache/lucene/index/index.40.optimized.nocfs.zip (working copy) Property changes on: lucene/core/src/test/org/apache/lucene/index/index.40.optimized.nocfs.zip ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/core/src/test/org/apache/lucene/index/index.40.optimized.nocfs.zip:r1512937 Index: lucene/core/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java =================================================================== --- lucene/core/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java (revision 1512948) +++ lucene/core/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java (working copy) Property changes on: lucene/core/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/core/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java:r1512937 Index: lucene/core/src/test/org/apache/lucene/search/TestSort.java =================================================================== --- lucene/core/src/test/org/apache/lucene/search/TestSort.java (revision 1512948) +++ lucene/core/src/test/org/apache/lucene/search/TestSort.java (working copy) Property changes on: lucene/core/src/test/org/apache/lucene/search/TestSort.java ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/core/src/test/org/apache/lucene/search/TestSort.java:r1512937 Index: lucene/core/src/test/org/apache/lucene/search/TestSortDocValues.java =================================================================== --- lucene/core/src/test/org/apache/lucene/search/TestSortDocValues.java (revision 1512948) +++ lucene/core/src/test/org/apache/lucene/search/TestSortDocValues.java (working copy) Property changes on: lucene/core/src/test/org/apache/lucene/search/TestSortDocValues.java ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/core/src/test/org/apache/lucene/search/TestSortDocValues.java:r1512937 Index: lucene/core/src/test/org/apache/lucene/search/TestSortRandom.java =================================================================== --- lucene/core/src/test/org/apache/lucene/search/TestSortRandom.java (revision 1512948) +++ lucene/core/src/test/org/apache/lucene/search/TestSortRandom.java (working copy) Property changes on: lucene/core/src/test/org/apache/lucene/search/TestSortRandom.java ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/core/src/test/org/apache/lucene/search/TestSortRandom.java:r1512937 Index: lucene/core/src/test/org/apache/lucene/search/TestTopFieldCollector.java =================================================================== --- lucene/core/src/test/org/apache/lucene/search/TestTopFieldCollector.java (revision 1512948) +++ lucene/core/src/test/org/apache/lucene/search/TestTopFieldCollector.java (working copy) Property changes on: lucene/core/src/test/org/apache/lucene/search/TestTopFieldCollector.java ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/core/src/test/org/apache/lucene/search/TestTopFieldCollector.java:r1512937 Index: lucene/core/src/test/org/apache/lucene/search/TestTotalHitCountCollector.java =================================================================== --- lucene/core/src/test/org/apache/lucene/search/TestTotalHitCountCollector.java (revision 1512948) +++ lucene/core/src/test/org/apache/lucene/search/TestTotalHitCountCollector.java (working copy) Property changes on: lucene/core/src/test/org/apache/lucene/search/TestTotalHitCountCollector.java ___________________________________________________________________ Modified: svn:mergeinfo Merged /lucene/dev/trunk/lucene/core/src/test/org/apache/lucene/search/TestTotalHitCountCollector.java:r1512937 Index: lucene/core/src/test/org/apache/lucene/store/TestBufferedIndexInput.java =================================================================== --- lucene/core/src/test/org/apache/lucene/store/TestBufferedIndexInput.java (revision 1512948) +++ lucene/core/src/test/org/apache/lucene/store/TestBufferedIndexInput.java (working copy) @@ -81,24 +81,6 @@ public void testReadBytes() throws Exception { MyBufferedIndexInput input = new MyBufferedIndexInput(); runReadBytes(input, BufferedIndexInput.BUFFER_SIZE, random()); - - // This tests the workaround code for LUCENE-1566 where readBytesInternal - // provides a workaround for a JVM Bug that incorrectly raises a OOM Error - // when a large byte buffer is passed to a file read. - // NOTE: this does only test the chunked reads and NOT if the Bug is triggered. - //final int tmpFileSize = 1024 * 1024 * 5; - final int inputBufferSize = 128; - File tmpInputFile = _TestUtil.createTempFile("IndexInput", "tmpFile", TEMP_DIR); - tmpInputFile.deleteOnExit(); - writeBytes(tmpInputFile, TEST_FILE_LENGTH); - - // run test with chunk size of 10 bytes - runReadBytesAndClose(new SimpleFSIndexInput("SimpleFSIndexInput(path=\"" + tmpInputFile + "\")", tmpInputFile, - newIOContext(random()), 10), inputBufferSize, random()); - - // run test with chunk size of 10 bytes - runReadBytesAndClose(new NIOFSIndexInput(tmpInputFile, - newIOContext(random()), 10), inputBufferSize, random()); } private void runReadBytesAndClose(IndexInput input, int bufferSize, Random r) @@ -208,6 +190,7 @@ private static byte byten(long n){ return (byte)(n*n%256); } + private static class MyBufferedIndexInput extends BufferedIndexInput { private long pos; private long len; Index: lucene/core/src/test/org/apache/lucene/store/TestDirectory.java =================================================================== --- lucene/core/src/test/org/apache/lucene/store/TestDirectory.java (revision 1512948) +++ lucene/core/src/test/org/apache/lucene/store/TestDirectory.java (working copy) @@ -133,52 +133,66 @@ // Test that different instances of FSDirectory can coexist on the same // path, can read, write, and lock files. public void testDirectInstantiation() throws Exception { - File path = _TestUtil.getTempDir("testDirectInstantiation"); + final File path = _TestUtil.getTempDir("testDirectInstantiation"); + + final byte[] largeBuffer = new byte[random().nextInt(256*1024)], largeReadBuffer = new byte[largeBuffer.length]; + for (int i = 0; i < largeBuffer.length; i++) { + largeBuffer[i] = (byte) i; // automatically loops with modulo + } - int sz = 3; - Directory[] dirs = new Directory[sz]; + final FSDirectory[] dirs = new FSDirectory[] { + new SimpleFSDirectory(path, null), + new NIOFSDirectory(path, null), + new MMapDirectory(path, null) + }; - dirs[0] = new SimpleFSDirectory(path, null); - dirs[1] = new NIOFSDirectory(path, null); - dirs[2] = new MMapDirectory(path, null); - - for (int i=0; i