[LUCENE-9309] IW#addIndices(CodecReader) might delete files concurrently to IW#rollback

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 9.0, 8.6
• Lucene Fields: New, Patch Available
• Assignee: Simon Willnauer
• Reporter: Simon Willnauer
• Created: 08/Apr/20 09:58
• Updated: 28/Aug/22 16:01
• Resolved: 10/Apr/20 11:24

Description

During work on LUCENE-9304 mikemccand ran into a failure:

org.apache.lucene.index.TestAddIndexes > test suite's output saved to /home/mike/src/simon/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.index.TestAddIndexes.txt, copied below:
   >     java.nio.file.NoSuchFileException: _gt_Lucene85FieldsIndex-doc_ids_6u.tmp
   >         at __randomizedtesting.SeedInfo.seed([4760FA81FBD4B2CE:A147156E5F7BD9B0]:0)
   >         at org.apache.lucene.store.ByteBuffersDirectory.deleteFile(ByteBuffersDirectory.java:148)
   >         at org.apache.lucene.store.MockDirectoryWrapper.deleteFile(MockDirectoryWrapper.java:607)
   >         at org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
   >         at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696)
   >         at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690)
   >         at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:449)
   >         at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2334)
   >         at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2275)
   >         at org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2268)
   >         at org.apache.lucene.index.TestAddIndexes.testAddIndexesWithRollback(TestAddIndexes.java:974)

  2> NOTE: reproduce with: ant test  -Dtestcase=TestAddIndexes -Dtests.method=testAddIndexesWithRollback -Dtests.seed=4760FA81FBD4B2CE -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=fr-GP -Dtests.t\
imezone=Asia/Tbilisi -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> NOTE: test params are: codec=Asserting(Lucene84): {c=PostingsFormat(name=LuceneFixedGap), id=PostingsFormat(name=LuceneFixedGap), f1=PostingsFormat(name=LuceneFixedGap), f2=BlockTreeOrds(blocksize=128)\
, version=BlockTreeOrds(blocksize=128), content=FST50}, docValues:{dv=DocValuesFormat(name=Lucene80), soft_delete=DocValuesFormat(name=Lucene80), doc=DocValuesFormat(name=Lucene80), id=DocValuesFormat(name=\
Asserting), content=DocValuesFormat(name=Asserting), doc2d=DocValuesFormat(name=Lucene80)}, maxPointsInLeafNode=982, maxMBSortInHeap=5.837219998050092, sim=Asserting(org.apache.lucene.search.similarities.As\
sertingSimilarity@6ce38471), locale=fr-GP, timezone=Asia/Tbilisi

While this unfortunately doesn't reproduce it's likely a bug that exists for quite some time but never showed up until LUCENE-9147 which uses a temporary output. That's fine but with IW#addIndices(CodecReader...) not registering the merge it does in the IW we never wait for the merge to finish while rollback and if that merge finishes concurrently it will also remove these .tmp files.

There are many ways to fix this and I can work on a patch, but hey do we really need to be able to add indices while we index and do that on an open and live IW or can it be a tool on top of it?

Issue Links

• links to: GitHub Pull Request #1418