[ASTERIXDB-3394] Introducing concurrent size bound merge policy

• Type: Improvement
• Status: In Review
• Priority: Major
• Resolution: Unresolved
• Component/s: STO - Storage
• Labels: triaged
• Assignee: Unassigned
• Reporter: Ritik Raj
• Created: 03/May/24 21:00
• Updated: 22/Sep/24 16:54

Description

Introducing size-bound merge policy to account for cloud service providers' max blob storage file size and to prevent exceeding that specified limit.

• Add storage config to set max mergable component size.
• Add a new merge policy and make it our default.
• The new merge policy will schedule merges similar to the current ConcurrentMergePolicy but it will consider the max mergable component size.
• Update the isMergeLagging check on max components to use the "effective" meragable components which can actually be merged rather than the current total number of disk components.
• In case of max components is reached and no merges could be scheduled, force a merge on the effective meragable components.