[DRILL-6180] Use System Option "output_batch_size" for External Sort

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Fix Version/s: 1.13.0
• Component/s: Execution - Flow
• Labels: ready-to-commit
• Assignee: Padma Penumarthy
• Reporter: Padma Penumarthy
• Reviewer: Paul Rogers
• Created: 22/Feb/18 19:55
• Updated: 14/Mar/18 04:53
• Resolved: 03/Mar/18 18:49

Description

External Sort has boot time configuration for output batch size "drill.exec.sort.external.spill.merge_batch_size" which is defaulted to 16M.

To make batch sizing configuration uniform across all operators, change this to use new system option that is added "drill.exec.memory.operator.output_batch_size". This option has default value of 32M.

So, what are the implications if default is changed to 32M for external sort ?

Instead, should we change the output batch size default to 16M for all operators ?

 

 

Issue Links

• relates to: DRILL-6238 Batch sizing for operators
• links to: GitHub Pull Request #1129