Lucene - Core
  1. Lucene - Core
  2. LUCENE-2573

Tiered flushing of DWPTs by RAM with low/high water marks

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Realtime Branch
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Now that we have DocumentsWriterPerThreads we need to track total consumed RAM across all DWPTs.

      A flushing strategy idea that was discussed in LUCENE-2324 was to use a tiered approach:

      • Flush the first DWPT at a low water mark (e.g. at 90% of allowed RAM)
      • Flush all DWPTs at a high water mark (e.g. at 110%)
      • Use linear steps in between high and low watermark: E.g. when 5 DWPTs are used, flush at 90%, 95%, 100%, 105% and 110%.

      Should we allow the user to configure the low and high water mark values explicitly using total values (e.g. low water mark at 120MB, high water mark at 140MB)? Or shall we keep for simplicity the single setRAMBufferSizeMB() config method and use something like 90% and 110% for the water marks?

      1. LUCENE-2573.patch
        104 kB
        Simon Willnauer
      2. LUCENE-2573.patch
        85 kB
        Simon Willnauer
      3. LUCENE-2573.patch
        69 kB
        Simon Willnauer
      4. LUCENE-2573.patch
        71 kB
        Simon Willnauer
      5. LUCENE-2573.patch
        61 kB
        Simon Willnauer
      6. LUCENE-2573.patch
        58 kB
        Simon Willnauer
      7. LUCENE-2573.patch
        19 kB
        Jason Rutherglen
      8. LUCENE-2573.patch
        19 kB
        Jason Rutherglen
      9. LUCENE-2573.patch
        17 kB
        Jason Rutherglen

        Issue Links

          Activity

          Michael Busch created issue -
          Michael Busch made changes -
          Field Original Value New Value
          Link This issue blocks LUCENE-2571 [ LUCENE-2571 ]
          Michael Busch made changes -
          Link This issue is related to LUCENE-2324 [ LUCENE-2324 ]
          Jason Rutherglen made changes -
          Attachment LUCENE-2573.patch [ 12453960 ]
          Jason Rutherglen made changes -
          Attachment LUCENE-2573.patch [ 12454034 ]
          Jason Rutherglen made changes -
          Attachment LUCENE-2573.patch [ 12454055 ]
          Robert Muir made changes -
          Component/s Index [ 12310232 ]
          Mark Thomas made changes -
          Workflow jira [ 12516763 ] Default workflow, editable Closed status [ 12563744 ]
          Mark Thomas made changes -
          Workflow Default workflow, editable Closed status [ 12563744 ] jira [ 12584413 ]
          Simon Willnauer made changes -
          Assignee Michael Busch [ michaelbusch ] Simon Willnauer [ simonw ]
          Simon Willnauer made changes -
          Attachment LUCENE-2573.patch [ 12472834 ]
          Simon Willnauer made changes -
          Attachment LUCENE-2573.patch [ 12472932 ]
          Simon Willnauer made changes -
          Attachment LUCENE-2573.patch [ 12473580 ]
          Simon Willnauer made changes -
          Attachment LUCENE-2573.patch [ 12474007 ]
          Simon Willnauer made changes -
          Attachment LUCENE-2573.patch [ 12474404 ]
          Simon Willnauer made changes -
          Attachment LUCENE-2573.patch [ 12474885 ]
          Simon Willnauer made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              Simon Willnauer
              Reporter:
              Michael Busch
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development