[KYLIN-2929] speed up Dump file performance

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v2.0.0
• Fix Version/s: v2.3.0
• Component/s: Query Engine
• Labels: Performance
• Assignee: fengYu
• Reporter: fengYu
• Created: 11/Oct/17 11:08
• Updated: 02/Apr/18 13:42
• Resolved: 04/Feb/18 15:36

Description

when I work on KYLIN-2926, I find coprocessor will dump to disk once estimatedMemSize is bigger than spillThreshold, and found that spill data size is extraordinary smaller that estimatedMemSize, in my case dump file size is about 8MB and spillThreshold is setting to 3GB.

So, I try to keep the spill data in memory rather than write the file to disk immediately, and when those in-memory spill data reach the threshold, write all spill files together.

In my case, the coprocessor process cost time drop from 22s to 16s, it is about 30% upgrade。