Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-9511

Include StoredFieldsWriter in DWPT accounting

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 8.7
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      StoredFieldsWriter might consume some heap space memory that can have a significant impact on decisions made in the IW if writers should be stalled or DWPTs should be flushed if memory settings are small in IWC and flushes are frequent. We should add some accounting to the StoredFieldsWriter since it's part of the DWPT lifecycle and not just present during flush.

      Our nightly builds ran into some OOMs due to the large chunk size used in the CompressedStoredFieldsFormat. The reason are very frequent flushes due to small maxBufferedDocs which causes 300+ DWPTs to be blocked for flush causing ultimately an OOM exception.

       
       NOTE: reproduce with: ant test  -Dtestcase=TestIndexingSequenceNumbers -Dtests.method=testStressConcurrentCommit -Dtests.seed=A04943A98C8E2954 -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=vo-001 -Dtests.timezone=Africa/Ouagadougou -Dtests.asserts=true -Dtests.file.encoding=UTF8*06:06:15*    [junit4] ERROR    107s J3 | TestIndexingSequenceNumbers.testStressConcurrentCommit <<<*06:06:15*    [junit4]    > Throwable #1: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed*06:06:15*    [junit4]    > 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:876)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:890)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3727)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit(TestIndexingSequenceNumbers.java:228)*06:06:15*    [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*06:06:15*    [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)*06:06:15*    [junit4]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*06:06:15*    [junit4]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)*06:06:15*    [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:834)*06:06:15*    [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space*06:06:15*    [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A04943A98C8E2954]:0)*06:06:15*    [junit4]    > 	at org.apache.lucene.store.GrowableByteArrayDataOutput.<init>(GrowableByteArrayDataOutput.java:46)*06:06:15*    [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:111)*06:06:15*    [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:130)*06:06:15*    [junit4]    > 	at org.apache.lucene.codecs.lucene87.Lucene87StoredFieldsFormat.fieldsWriter(Lucene87StoredFieldsFormat.java:141)*06:06:15*    [junit4]    > 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:48)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:39)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:46)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:426)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:462)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:233)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:419)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1470)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1463)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:206)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1140, name=Thread-1095, state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]*06:06:15*    [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1139, name=Thread-1094, state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]*06:06:15*    [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space*06:06:15*    [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A04943A98C8E2954]:0)*06:06:15*    [junit4]    > 	at org.apache.lucene.store.GrowableByteArrayDataOutput.<init>(GrowableByteArrayDataOutput.java:46)*06:06:15*    [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:111)*06:06:15*    [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:130)*06:06:15*    [junit4]    > 	at org.apache.lucene.codecs.lucene87.Lucene87StoredFieldsFormat.fieldsWriter(Lucene87StoredFieldsFormat.java:141)*06:06:15*    [junit4]    > 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:48)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:39)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:46)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:426)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:462)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:233)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:419)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1470)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1463)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:206)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1137, name=Thread-1092, state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]*06:06:15*    [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space*06:06:15*    [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A04943A98C8E2954]:0)*06:06:15*    [junit4]    > 	at org.apache.lucene.store.GrowableByteArrayDataOutput.<init>(GrowableByteArrayDataOutput.java:46)*06:06:15*    [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:111)*06:06:15*    [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:130)*06:06:15*    [junit4]    > 	at org.apache.lucene.codecs.lucene87.Lucene87StoredFieldsFormat.fieldsWriter(Lucene87StoredFieldsFormat.java:141)*06:06:15*    [junit4]    > 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:48)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:39)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:46)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:426)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:462)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:233)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:419)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1470)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1463)*06:06:15*    [junit4]    > 	at org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:206)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1141, name=Thread-1096, state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]*06:06:15*    [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1138, name=Thread-1093, state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]*06:06:15*    [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space*06:06:15*    [junit4]   2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/apache+lucene-solr+nightly+branch_8x/lucene/build/core/test/J3/temp/lucene.index.TestIndexingSequenceNumbers_A04943A98C8E2954-001*06:06:15*    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene87): \{id=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:\{thread=DocValuesFormat(name=Lucene80), ___soft_deletes=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=837, maxMBSortInHeap=5.048657222730272, sim=Asserting(RandomSimilarity(queryNorm=true): \{id=IB SPL-D2}), locale=vo-001, timezone=Africa/Ouagadougou*06:06:15*    [junit4]   2> NOTE: Linux 4.12.14-122.32-default amd64/Oracle Corporation 11.0.2 (64-bit)/cpus=32,threads=1,free=444391792,total=536870912*06:06:15*    [junit4]   2> NOTE: All tests run in this JVM: [TestApproximationSearchEquivalence, TestPositionIncrement, TestCachingTokenFilter, TestSimpleExplanationsOfNonMatches, TestPostingsOffsets, TestTimSorter, TestSpanNearQuery, TestConstantScoreScorer, TestSpanExplanations, TestIndexingSequenceNumbers]
       
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                simonw Simon Willnauer
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m