Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Implemented
-
None
-
None
-
None
-
None
Description
The default PrefixMergePolicy has a number of problems: it only schedules 1 merge at a time; it stops merging components when they are too large. Both problems will lead to undesirable performance behavior. This issue introduces a ConcurrentMergePolicy that fixes these problems.
It has the following four parameters:
- minComponentMergeCount: the minimum number of components to trigger a merge
- maxComponentMergeCount: the maximum number of components per merge
- maxComponentCount: the maximum number of components tolerated in total; when this number is reached, flush will be stopped
- sizeRatio: a merge is scheduled if the size of the oldest component <= sizeRatio * total size of youngest components. A larger size ratio(>1) implies fewer merges, but this leads to more components. A smaller size ratio (<1) implies more merges, but this improves query performance and space utilization.
Concurrent merges will be scheduled as well. Given a sequence of components ordered by newest to oldest, this policy first finds a longest prefix of this sequence so that no component is being merged. The merge decision will then be based on this prefix.