Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2898

CMS merge throttling is not aggressive enough

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1, 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I hit this crab while working on the NRT benchmarker (in luceneutil).

      CMS today forcefully idles any incoming threads, when there are too many merges pending.

      This is the last line of defense that it has, since it also juggles thread priorities (and forcefully idles the biggest merges) to try to reduce the outstanding merge count.

      But when it cannot keep up it has no choice but to stall those threads responsible for making new segments.

      However, the logic is in the wrong place now – the stalling happens after pulling the next merge from IW. This is poor because it means if you have N indexing threads, you allow max + N outstanding merges.

      I have a simple fix, which is to just move the stall logic to before we pull the next merge from IW.

        Attachments

          Activity

            People

            • Assignee:
              mikemccand Michael McCandless
              Reporter:
              mikemccand Michael McCandless
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: