[LUCENE-8328] ReadersAndUpdates#getLatestReader should execute under lock

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 7.4, 8.0
• Lucene Fields: Patch Available
• Assignee: Unassigned
• Reporter: Nhat Nguyen
• Created: 22/May/18 20:59
• Updated: 28/Aug/22 15:30
• Resolved: 23/May/18 05:59

Description

It's possible for a merge thread to acquire an index reader which is closed before it can incRef.

Merge stack trace:

at __randomizedtesting.SeedInfo.seed([136983A068AA2F9D]:0)
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:257)
at org.apache.lucene.index.IndexReader.incRef(IndexReader.java:184)
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:198)
at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:728)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4355)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4043)
at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2145)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:542)
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:288)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:263)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:253)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)

Refresh stack trace:
at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:238)
at org.apache.lucene.index.ReadersAndUpdates.createNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:675)
at org.apache.lucene.index.ReadersAndUpdates.swapNewReaderWithLatestLiveDocs(ReadersAndUpdates.java:686)
at org.apache.lucene.index.ReadersAndUpdates.getLatestReader(ReadersAndUpdates.java:260)
at org.elasticsearch.index.shard.ElasticsearchMergePolicy.keepFullyDeletedSegment(ElasticsearchMergePolicy.java:143)
at org.apache.lucene.index.ReadersAndUpdates.keepFullyDeletedSegment(ReadersAndUpdates.java:769)
at org.apache.lucene.index.IndexWriter.isFullyDeleted(IndexWriter.java:5124)
at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3306)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:514)
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:288)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:263)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:253)
at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)

The problem is that `ReadersAndUpdates#getLatestReader` is executed concurrently without holding lock.