Details
-
Improvement
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
-
New
Description
If you are doing lots of concurrent indexing, NRT search regularly publishes many tiny segments, which in-turn pushes a lot of pressure on merging, which needs to constantly merge these tiny segments so that the total number of segments of the index remains under the budget.
In parallel, TieredMergePolicy's behavior is to merge aggressively segments that are below the floor size. The budget of the number of segments allowed in the index is computed as if all segments were larger than the floor size, and merges that only contain segments below the floor size get a perfect skew which guarantees them to get a better score than any merge that contains one or more segments above the floor size.
I'm considering reducing the merging overhead in the NRT case by raising maxMergeAtOnce and allowing merges to merge more than mergeFactor segments as long as the number of merged segments is below maxMergeAtOnce and the merged segment size is below the floor segment size.
Said otherwise, "normal" merges would be allowed to merge up to mergeFactor segments like today, while small merges (size of the merged segment < floor segment bytes) could go up to maxMergeAtOnce.
Attachments
Issue Links
- links to
