[DRILL-6123] Limit batch size for Merge Join based on memory

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Fix Version/s: 1.13.0
• Component/s: Execution - Flow
• Labels: ready-to-commit
• Assignee: Padma Penumarthy
• Reporter: Padma Penumarthy
• Reviewer: Paul Rogers
• Created: 31/Jan/18 00:19
• Updated: 21/Mar/18 20:39
• Resolved: 19/Feb/18 09:41

Description

Merge join limits output batch size to 32K rows irrespective of row size. This can create very large or very small batches (in terms of memory), depending upon average row width. Change this to figure out output row count based on memory specified with the new outputBatchSize option and average row width of incoming left and right batches. Output row count will be minimum of 1 and max of 64k. 

Issue Links

• relates to: DRILL-6238 Batch sizing for operators
• links to: GitHub Pull Request #1107