Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2573

Tiered flushing of DWPTs by RAM with low/high water marks

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Realtime Branch
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Now that we have DocumentsWriterPerThreads we need to track total consumed RAM across all DWPTs.

      A flushing strategy idea that was discussed in LUCENE-2324 was to use a tiered approach:

      • Flush the first DWPT at a low water mark (e.g. at 90% of allowed RAM)
      • Flush all DWPTs at a high water mark (e.g. at 110%)
      • Use linear steps in between high and low watermark: E.g. when 5 DWPTs are used, flush at 90%, 95%, 100%, 105% and 110%.

      Should we allow the user to configure the low and high water mark values explicitly using total values (e.g. low water mark at 120MB, high water mark at 140MB)? Or shall we keep for simplicity the single setRAMBufferSizeMB() config method and use something like 90% and 110% for the water marks?

        Attachments

        1. LUCENE-2573.patch
          17 kB
          Jason Rutherglen
        2. LUCENE-2573.patch
          19 kB
          Jason Rutherglen
        3. LUCENE-2573.patch
          19 kB
          Jason Rutherglen
        4. LUCENE-2573.patch
          58 kB
          Simon Willnauer
        5. LUCENE-2573.patch
          61 kB
          Simon Willnauer
        6. LUCENE-2573.patch
          71 kB
          Simon Willnauer
        7. LUCENE-2573.patch
          69 kB
          Simon Willnauer
        8. LUCENE-2573.patch
          85 kB
          Simon Willnauer
        9. LUCENE-2573.patch
          104 kB
          Simon Willnauer

          Issue Links

            Activity

              People

              • Assignee:
                simonw Simon Willnauer
                Reporter:
                michaelbusch Michael Busch
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: