Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-9535

Investigate recent indexing slowdown for wikimedium documents


    • Task
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 8.7
    • None
    • None
    • New


      Nightly benchmarks report a ~10% slowdown for 1kB documents as of September 9th: http://people.apache.org/~mikemccand/lucenebench/indexing.html.

      On that day, we added stored fields in DWPT accounting (LUCENE-9511), so I first thought this could be due to smaller flushed segments and more merging, but I still wonder whether there's something else. The benchmark runs with 8GB of heap, 2GB of RAM buffer and 36 indexing threads. So it's about 2GB/36 = 57MB of RAM buffer per thread in the worst-case scenario that all DWPTs get full at the same time. Stored fields account for about 0.7MB of memory, or 1% of the indexing buffer size. How can a 1% reduction of buffering capacity explain a 10% indexing slowdown? I looked into this further by running indexing benchmarks locally with 8 indexing threads and 128MB of indexing buffer memory, which would make this issue even more apparent if the smaller RAM buffer was the cause, but I'm not seeing a regression and actually I'm seeing similar number of flushes when I disabled memory accounting for stored fields.

      I ran indexing under a profiler to see whether something else could cause this slowdown, e.g. slow implementations of ramBytesUsed on stored fields writers, but nothing surprising showed up and the profile looked just like I would have expected.

      Another question I have is why the 4kB benchmark is not affected at all.


        1. cpu_profile.svg
          384 kB
          Adrien Grand



            Unassigned Unassigned
            jpountz Adrien Grand
            0 Vote for this issue
            5 Start watching this issue



              Time Tracking

                Original Estimate - Not Specified
                Not Specified
                Remaining Estimate - 0h
                Time Spent - 5h 40m
                5h 40m