Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7868

Use multiple threads to apply deletes and DV updates


    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.0
    • Component/s: None
    • Labels:
    • Lucene Fields:


      Today, when users delete documents or apply doc values updates, IndexWriter buffers them up into frozen packets and then eventually uses a single thread (BufferedUpdatesStream.applyDeletesAndUpdates) to resolve delete/update terms to docids. This thread also holds IW's monitor lock, so it also blocks refresh, merges starting/finishing, commits, etc.

      We have heavily optimized this part of Lucene over time, e.g. LUCENE-6161, LUCENE-2897, LUCENE-2680, LUCENE-3342, but still, it's a single thread so it can't use multiple CPU cores commonly available now.

      This doesn't affect append-only usage, but for update-heavy users (me!) this can be a big bottleneck, and causes long stop-the-world hangs during indexing.

      I have an initial exploratory patch to make these lookups concurrent, without holding IW's lock, so that when a new packet of deletes is pushed, which happens when we flush a new segment, we immediately use that same indexing thread to and resolve the deletions.

      This is analogous to when we made segment flushing concurrent (LUCENE-3023), just for deletes and DV updates as well.


        1. LUCENE-7868.patch
          190 kB
          Michael McCandless
        2. cpu-after.png
          94 kB
          Michael McCandless
        3. cpu-before.png
          85 kB
          Michael McCandless
        4. LUCENE-7868.patch
          279 kB
          Michael McCandless
        5. LUCENE-7868.patch
          295 kB
          Michael McCandless
        6. LUCENE-7868.patch
          333 kB
          Michael McCandless
        7. LUCENE-7868.patch
          336 kB
          Michael McCandless
        8. LUCENE-7868.patch
          336 kB
          Michael McCandless



            • Assignee:
              mikemccand Michael McCandless
              mikemccand Michael McCandless
            • Votes:
              0 Vote for this issue
              5 Start watching this issue


              • Created: