Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5654

CompoundFileWriter.close suppresses OOME

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.8.1, 4.9, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I'm working on a test case, and it hit OOME while writing the compound
      file; the test itself is separately buggy, but what was confounding
      was that CompoundFileWriter continued trying to write to the
      RAMOutputStream even after it had already hit OOME.

      RAMOutputStream could be better here (e.g. only increment
      currentBufferIndex if switchCurrentBuffer succeeds), but also I think
      we should fix CompoundFileWriter.close to .closeWhileHandlingExc even
      on non-IOExc.

      It results in scary looking excs like this:

      java.lang.RuntimeException: java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: Index: 2428, Size: 2427
      	at __randomizedtesting.SeedInfo.seed([4E313C43B4C1223B]:0)
      	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:222)
      Caused by: java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: Index: 2428, Size: 2427
      	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$CheckSegmentCount.run(TestIndexWriterThreadsToSegments.java:137)
      	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:213)
      	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:355)
      	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:219)
      Caused by: java.lang.IndexOutOfBoundsException: Index: 2428, Size: 2427
      	at java.util.ArrayList.rangeCheck(ArrayList.java:604)
      	at java.util.ArrayList.get(ArrayList.java:382)
      	at org.apache.lucene.store.RAMFile.getBuffer(RAMFile.java:68)
      	at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:154)
      	at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:138)
      	at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:125)
      	at org.apache.lucene.store.MockIndexOutputWrapper.writeByte(MockIndexOutputWrapper.java:111)
      	at org.apache.lucene.store.DataOutput.writeInt(DataOutput.java:70)
      	at org.apache.lucene.codecs.CodecUtil.writeFooter(CodecUtil.java:184)
      	at org.apache.lucene.store.CompoundFileWriter.close(CompoundFileWriter.java:144)
      	at org.apache.lucene.store.CompoundFileDirectory.close(CompoundFileDirectory.java:178)
      	at org.apache.lucene.util.IOUtils.closeWhileHandlingException(IOUtils.java:88)
      	at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4512)
      	at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:485)
      	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:452)
      	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:518)
      	at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:629)
      	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:376)
      	at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:285)
      	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:260)
      	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:250)
      	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:124)
      	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$CheckSegmentCount.run(TestIndexWriterThreadsToSegments.java:124)
      

        Attachments

          Activity

            People

            • Assignee:
              rcmuir Robert Muir
              Reporter:
              mikemccand Michael McCandless
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: