[DRILL-6113] Limit batch size for Merge Receiver

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.12.0
• Component/s: Execution - Flow
• Assignee: Padma Penumarthy
• Reporter: Padma Penumarthy
• Created: 26/Jan/18 22:20
• Updated: 30/May/18 20:47

Description

Merge receiver has hard coded limit of 32K rows as batch. Since rows can be of varying width, it is difficult to predict the output batch size (in terms of memory) for this operator. Change this to derive row count based on actual memory available. We are introducing a new option called outputBatchSize to limit the batch size of each operator. Use the memory configured from that. Figure out the average row width of outgoing batch based on averages of batches coming from incoming streams. Limit the row count based on memory available to use and average row width.

 

Issue Links

• relates to: DRILL-6238 Batch sizing for operators