Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-9309

IW#addIndices(CodecReader) might delete files concurrently to IW#rollback

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: master (9.0), 8.6
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      During work on LUCENE-9304 Michael McCandless ran into a failure:

      org.apache.lucene.index.TestAddIndexes > test suite's output saved to /home/mike/src/simon/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.index.TestAddIndexes.txt, copied below:
         >     java.nio.file.NoSuchFileException: _gt_Lucene85FieldsIndex-doc_ids_6u.tmp
         >         at __randomizedtesting.SeedInfo.seed([4760FA81FBD4B2CE:A147156E5F7BD9B0]:0)
         >         at org.apache.lucene.store.ByteBuffersDirectory.deleteFile(ByteBuffersDirectory.java:148)
         >         at org.apache.lucene.store.MockDirectoryWrapper.deleteFile(MockDirectoryWrapper.java:607)
         >         at org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
         >         at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696)
         >         at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690)
         >         at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:449)
         >         at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2334)
         >         at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2275)
         >         at org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2268)
         >         at org.apache.lucene.index.TestAddIndexes.testAddIndexesWithRollback(TestAddIndexes.java:974)
      
        2> NOTE: reproduce with: ant test  -Dtestcase=TestAddIndexes -Dtests.method=testAddIndexesWithRollback -Dtests.seed=4760FA81FBD4B2CE -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=fr-GP -Dtests.t\
      imezone=Asia/Tbilisi -Dtests.asserts=true -Dtests.file.encoding=UTF-8
        2> NOTE: test params are: codec=Asserting(Lucene84): {c=PostingsFormat(name=LuceneFixedGap), id=PostingsFormat(name=LuceneFixedGap), f1=PostingsFormat(name=LuceneFixedGap), f2=BlockTreeOrds(blocksize=128)\
      , version=BlockTreeOrds(blocksize=128), content=FST50}, docValues:{dv=DocValuesFormat(name=Lucene80), soft_delete=DocValuesFormat(name=Lucene80), doc=DocValuesFormat(name=Lucene80), id=DocValuesFormat(name=\
      Asserting), content=DocValuesFormat(name=Asserting), doc2d=DocValuesFormat(name=Lucene80)}, maxPointsInLeafNode=982, maxMBSortInHeap=5.837219998050092, sim=Asserting(org.apache.lucene.search.similarities.As\
      sertingSimilarity@6ce38471), locale=fr-GP, timezone=Asia/Tbilisi
      

      While this unfortunately doesn't reproduce it's likely a bug that exists for quite some time but never showed up until LUCENE-9147 which uses a temporary output. That's fine but with IW#addIndices(CodecReader...) not registering the merge it does in the IW we never wait for the merge to finish while rollback and if that merge finishes concurrently it will also remove these .tmp files.

      There are many ways to fix this and I can work on a patch, but hey do we really need to be able to add indices while we index and do that on an open and live IW or can it be a tool on top of it?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                simonw Simon Willnauer
                Reporter:
                simonw Simon Willnauer
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m