Uploaded image for project: 'Apache AsterixDB'
  1. Apache AsterixDB
  2. ASTERIXDB-2600

Introduce ConcurrentMergePolicy

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The default PrefixMergePolicy has a number of problems: it only schedules 1 merge at a time; it stops merging components when they are too large. Both problems will lead to undesirable performance behavior. This issue introduces a ConcurrentMergePolicy that fixes these problems.

      It has the following four parameters:

      • minComponentMergeCount: the minimum number of components to trigger a merge
      • maxComponentMergeCount: the maximum number of components per merge
      • maxComponentCount: the maximum number of components tolerated in total; when this number is reached, flush will be stopped
      • sizeRatio: a merge is scheduled if the size of the oldest component <= sizeRatio * total size of youngest components. A larger size ratio(>1) implies fewer merges, but this leads to more components. A smaller size ratio (<1) implies more merges, but this improves query performance and space utilization.

      Concurrent merges will be scheduled as well. Given a sequence of components ordered by newest to oldest, this policy first finds a longest prefix of this sequence so that no component is being merged. The merge decision will then be based on this prefix.

        Attachments

          Activity

            People

            • Assignee:
              luochen01 Chen Luo
              Reporter:
              luochen01 Chen Luo
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: