Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-9189

TestIndexWriterDelete.testDeletesOnDiskFull can run for minutes

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 8.5
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I thought it was just the testUpdatesOnDiskFull, but looks like this one needs to be nightly too.

      Should look more into the test, but I know something causes it to make such an insane amount of files, that sorting them becomes a bottleneck.

      I guess also related is that it would be great if MockDirectoryWrapper's disk full check didn't trigger a sort of the files (via listAll): it does this check on like every i/o, would be nice for it to be less absurd. Maybe instead the test could check for disk full on not every i/o but some random sample of them?

      Temporarily lets make it nightly...

      PROFILE SUMMARY from 182501 samples
        tests.profile.count=10
        tests.profile.stacksize=1
        tests.profile.linenumbers=false
      PERCENT	SAMPLES	STACK
      15.89%	28995	java.lang.StringLatin1#compareTo()
      6.61%	12069	java.util.TimSort#mergeHi()
      5.96%	10878	java.util.TimSort#binarySort()
      3.41%	6231	java.util.concurrent.ConcurrentHashMap#tabAt()
      2.98%	5433	java.util.Comparators$NaturalOrderComparator#compare()
      2.12%	3876	org.apache.lucene.store.DataOutput#copyBytes()
      2.03%	3712	java.lang.String#compareTo()
      1.84%	3350	java.util.concurrent.ConcurrentHashMap#get()
      1.83%	3337	java.util.TimSort#mergeLo()
      1.67%	3047	java.util.ArrayList#add()
      

      All the file sorting is called from stacks like this, so its literally happening every writeByte() and so on

      0.73%	1329	java.util.TimSort#binarySort()
      			  at java.util.TimSort#sort()
      			  at java.util.Arrays#sort()
      			  at java.util.ArrayList#sort()
      			  at java.util.stream.SortedOps$RefSortingSink#end()
      			  at java.util.stream.AbstractPipeline#copyInto()
      			  at java.util.stream.AbstractPipeline#wrapAndCopyInto()
      			  at java.util.stream.AbstractPipeline#evaluate()
      			  at java.util.stream.AbstractPipeline#evaluateToArrayNode()
      			  at java.util.stream.ReferencePipeline#toArray()
      			  at org.apache.lucene.store.ByteBuffersDirectory#listAll()
      			  at org.apache.lucene.store.MockDirectoryWrapper#sizeInBytes()
      			  at org.apache.lucene.store.MockIndexOutputWrapper#checkDiskFull()
      			  at org.apache.lucene.store.MockIndexOutputWrapper#writeBytes()
      			  at org.apache.lucene.store.MockIndexOutputWrapper#writeByte()
      			  at org.apache.lucene.store.DataOutput#writeInt()
      			  at org.apache.lucene.codecs.CodecUtil#writeFooter()
      			  at org.apache.lucene.codecs.lucene50.Lucene50LiveDocsFormat#writeLiveDocs()
      			  at org.apache.lucene.codecs.asserting.AssertingLiveDocsFormat#writeLiveDocs()
      			  at org.apache.lucene.index.PendingDeletes#writeLiveDocs()
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rcmuir Robert Muir
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: