Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-8628

ByteBuffersDirectory merge exception

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 7.5
    • Fix Version/s: None
    • Component/s: core/store
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      An exception during merge for ByteBufferDirectory was reported on the lucene-dev mailinglist:

       

      Exception in thread "Lucene Merge Thread #879" org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception        at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:705)        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:685)Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception        at org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349)        at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669)        at org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664)        at org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5024)        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4539)        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4075)        at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:663)Caused by: java.lang.IllegalArgumentException: cannot write negative vLong (got: -4294878395)        at org.apache.lucene.store.DataOutput.writeVLong(DataOutput.java:225)        at org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.writeSkipData(Lucene50SkipWriter.java:180)        at org.apache.lucene.codecs.MultiLevelSkipListWriter.bufferSkip(MultiLevelSkipListWriter.java:143)        at org.apache.lucene.codecs.lucene50.Lucene50SkipWriter.bufferSkip(Lucene50SkipWriter.java:162)        at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:228)        at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)        at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)        at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4453)        ... 3 more

      Following the trace shows no apparent problems with the Lucene50SkipWripter. However, it uses getFilePointer to resolve the skip lengths, which for ByteBuffersDirectory ends in ByteBuffersDataOutput.size() with the code

      {{ public long size() {}}
      {{ long size = 0;}}
      {{ int blockCount = blocks.size();}}
      {{ if (blockCount >= 1) {}}
      {{ int fullBlockSize = (blockCount - 1) * blockSize();}}
      {{ int lastBlockSize = blocks.getLast().position();}}
      {{ size = fullBlockSize + lastBlockSize;}}
      {{ }}}
      {{ return size;}}
      {{ }}}

      One problem here is that

      int fullBlockSize = (blockCount - 1) * blockSize();

      can overflow. This should be changed to something like

      long fullBlockSize = (blockCount - 1) * (long)blockSize();

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              toke Toke Eskildsen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: