Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-6209

IndexWriter should confess when it stalls flushes

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 5.1, 6.0
    • None
    • None
    • New

    Description

      You tell IW how much RAM it's allowed to use to hold recently indexed documents before they must be written to disk.

      IW is willing to use up to 2X that amount for in-progress flushes.

      If the in-progress flushes go over that limit, then IW will stall them, hijacking indexing threads and having them wait until the in-progress flushes are below 2X indexing buffer size again.

      This is important back-pressure e.g. if you are indexing on a machine with many cores but slowish IO.

      Often when I profile an indexing heavy use case, even on fast IO (SSD) boxes, I see the methods associated with this back-pressure taking unexpected time ... yet IW never logs when it stalls/unstalls flushing. I think it should.

      Attachments

        1. LUCENE-6209.patch
          4 kB
          Michael McCandless

        Activity

          People

            mikemccand Michael McCandless
            mikemccand Michael McCandless
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: