Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-6209

IndexWriter should confess when it stalls flushes

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.1, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      You tell IW how much RAM it's allowed to use to hold recently indexed documents before they must be written to disk.

      IW is willing to use up to 2X that amount for in-progress flushes.

      If the in-progress flushes go over that limit, then IW will stall them, hijacking indexing threads and having them wait until the in-progress flushes are below 2X indexing buffer size again.

      This is important back-pressure e.g. if you are indexing on a machine with many cores but slowish IO.

      Often when I profile an indexing heavy use case, even on fast IO (SSD) boxes, I see the methods associated with this back-pressure taking unexpected time ... yet IW never logs when it stalls/unstalls flushing. I think it should.

        Attachments

        1. LUCENE-6209.patch
          4 kB
          Michael McCandless

          Activity

            People

            • Assignee:
              mikemccand Michael McCandless
              Reporter:
              mikemccand Michael McCandless
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: