[DRILL-6537] Limit the batch size for buffering operators based on how much memory they get

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.0
• Fix Version/s: 1.14.0
• Component/s: Execution - Relational Operators
• Labels: ready-to-commit
• Assignee: Padma Penumarthy
• Reporter: Padma Penumarthy
• Reviewer: Boaz Ben-Zvi
• Created: 25/Jun/18 20:20
• Updated: 02/Jul/18 06:33
• Resolved: 02/Jul/18 06:33

Description

we are using 16MB for output batch size for all operators by default. however, for buffering operators, depending upon how much memory they get allocated, it is possible that 16MB might be too much to use for output batch size, causing them to spill sometimes.

Have output batch size to be minimum of 16 MB or 20% of the allocated memory.

 

Issue Links

• links to: GitHub Pull Request #1342