[LUCENE-9534] DocumentsWriterFlushControl expects IndexingChain#ramBytesUsed to be thread-safe but it's not

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Lucene Fields: New
• Assignee: Simon Willnauer
• Reporter: Adrien Grand
• Created: 18/Sep/20 14:39
• Updated: 28/Aug/22 16:08
• Resolved: 18/Sep/20 16:05

Description

I found this after seeing the following test failure:

17:43:57    [junit4]   1> java.lang.NullPointerException
17:43:57    [junit4]   1> 	at org.apache.lucene.store.ByteBuffersDataOutput.lambda$ramBytesUsed$2(ByteBuffersDataOutput.java:405)
17:43:57    [junit4]   1> 	at java.base/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:229)
17:43:57    [junit4]   1> 	at java.base/java.util.ArrayDeque$DeqSpliterator.forEachRemaining(ArrayDeque.java:849)
17:43:57    [junit4]   1> 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
17:43:57    [junit4]   1> 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
17:43:57    [junit4]   1> 	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
17:43:57    [junit4]   1> 	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
17:43:57    [junit4]   1> 	at java.base/java.util.stream.LongPipeline.reduce(LongPipeline.java:474)
17:43:57    [junit4]   1> 	at java.base/java.util.stream.LongPipeline.sum(LongPipeline.java:432)
17:43:57    [junit4]   1> 	at org.apache.lucene.store.ByteBuffersDataOutput.ramBytesUsed(ByteBuffersDataOutput.java:405)
17:43:57    [junit4]   1> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.ramBytesUsed(CompressingStoredFieldsWriter.java:699)
17:43:57    [junit4]   1> 	at org.apache.lucene.index.DefaultIndexingChain.ramBytesUsed(DefaultIndexingChain.java:801)
17:43:57    [junit4]   1> 	at org.apache.lucene.index.DocumentsWriterPerThread.ramBytesUsed(DocumentsWriterPerThread.java:521)
17:43:57    [junit4]   1> 	at org.apache.lucene.index.DocumentsWriterFlushControl.findLargestNonPendingWriter(DocumentsWriterFlushControl.java:672)
17:43:57    [junit4]   1> 	at org.apache.lucene.index.FlushPolicy.findLargestNonPendingWriter(FlushPolicy.java:104)
17:43:57    [junit4]   1> 	at org.apache.lucene.index.FlushByRamOrCountsPolicy.markLargestWriterPending(FlushByRamOrCountsPolicy.java:93)
17:43:57    [junit4]   1> 	at org.apache.lucene.index.FlushByRamOrCountsPolicy.onInsert(FlushByRamOrCountsPolicy.java:82)
17:43:57    [junit4]   1> 	at org.apache.lucene.index.FlushPolicy.onUpdate(FlushPolicy.java:71)
17:43:57    [junit4]   1> 	at org.apache.lucene.index.DocumentsWriterFlushControl.doAfterDocument(DocumentsWriterFlushControl.java:178)
17:43:57    [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:430)
17:43:57    [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1471)
17:43:57    [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1757)
17:43:57    [junit4]   1> 	at org.apache.lucene.index.TestIndexWriterExceptions$IndexerThread.run(TestIndexWriterExceptions.java:216)
17:43:57    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterExceptions -Dtests.method=testRandomExceptionsThreads -Dtests.seed=3C0F2600B3EEA596 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=brx-IN -Dtests.timezone=Africa/Mbabane -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
17:43:57    [junit4] FAILURE 0.52s J3 | TestIndexWriterExceptions.testRandomExceptionsThreads <<<
17:43:57    [junit4]    > Throwable #1: java.lang.AssertionError: thread Indexer 3: hit unexpected failure
17:43:57    [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3C0F2600B3EEA596:83E16C1DBE01C597]:0)
17:43:57    [junit4]    > 	at org.apache.lucene.index.TestIndexWriterExceptions.testRandomExceptionsThreads(TestIndexWriterExceptions.java:316)
17:43:57    [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
17:43:57    [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
17:43:57    [junit4]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
17:43:57    [junit4]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
17:43:57    [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:834)

Issue Links

• links to: GitHub Pull Request #1889