Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7700

Move throughput control and merge aborting out of IndexWriter's core?

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.5, 7.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Here is a bit of a background:

      • I wanted to implement a custom merging strategy that would have a custom i/o flow control (global),
      • currently, the CMS is tightly coupled with a few classes – MergeRateLimiter, OneMerge, IndexWriter.

      Looking at the code it seems to me that everything with respect to I/O control could be nicely pulled out into classes that explicitly control the merging process, that is only MergePolicy and MergeScheduler. By default, one could even run without any additional I/O accounting overhead (which is currently in there, even if one doesn't use the CMS's throughput control).

      Such refactoring would also give a chance to nicely move things where they belong – job aborting into OneMerge (currently in RateLimiter), rate limiter lifecycle bound to OneMerge (MergeScheduler could then use per-merge or global accounting, as it pleases).

      Just a thought and some initial refactorings for discussion.

        Attachments

        1. LUCENE-7700.patch
          43 kB
          Dawid Weiss
        2. LUCENE-7700.patch
          42 kB
          Dawid Weiss
        3. LUCENE-7700.patch
          39 kB
          Dawid Weiss
        4. LUCENE-7700.patch
          16 kB
          Dawid Weiss

          Activity

            People

            • Assignee:
              dweiss Dawid Weiss
              Reporter:
              dweiss Dawid Weiss
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: