Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5654

CompoundFileWriter.close suppresses OOME

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 4.8.1, 4.9, 6.0
    • None
    • None
    • New

    Description

      I'm working on a test case, and it hit OOME while writing the compound
      file; the test itself is separately buggy, but what was confounding
      was that CompoundFileWriter continued trying to write to the
      RAMOutputStream even after it had already hit OOME.

      RAMOutputStream could be better here (e.g. only increment
      currentBufferIndex if switchCurrentBuffer succeeds), but also I think
      we should fix CompoundFileWriter.close to .closeWhileHandlingExc even
      on non-IOExc.

      It results in scary looking excs like this:

      java.lang.RuntimeException: java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: Index: 2428, Size: 2427
      	at __randomizedtesting.SeedInfo.seed([4E313C43B4C1223B]:0)
      	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:222)
      Caused by: java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: Index: 2428, Size: 2427
      	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$CheckSegmentCount.run(TestIndexWriterThreadsToSegments.java:137)
      	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:213)
      	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:355)
      	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:219)
      Caused by: java.lang.IndexOutOfBoundsException: Index: 2428, Size: 2427
      	at java.util.ArrayList.rangeCheck(ArrayList.java:604)
      	at java.util.ArrayList.get(ArrayList.java:382)
      	at org.apache.lucene.store.RAMFile.getBuffer(RAMFile.java:68)
      	at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:154)
      	at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:138)
      	at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:125)
      	at org.apache.lucene.store.MockIndexOutputWrapper.writeByte(MockIndexOutputWrapper.java:111)
      	at org.apache.lucene.store.DataOutput.writeInt(DataOutput.java:70)
      	at org.apache.lucene.codecs.CodecUtil.writeFooter(CodecUtil.java:184)
      	at org.apache.lucene.store.CompoundFileWriter.close(CompoundFileWriter.java:144)
      	at org.apache.lucene.store.CompoundFileDirectory.close(CompoundFileDirectory.java:178)
      	at org.apache.lucene.util.IOUtils.closeWhileHandlingException(IOUtils.java:88)
      	at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4512)
      	at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:485)
      	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:452)
      	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:518)
      	at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:629)
      	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:376)
      	at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:285)
      	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:260)
      	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:250)
      	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:124)
      	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$CheckSegmentCount.run(TestIndexWriterThreadsToSegments.java:124)
      

      Attachments

        1. LUCENE-5654.patch
          25 kB
          Robert Muir

        Activity

          People

            rcmuir Robert Muir
            mikemccand Michael McCandless
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment