[SPARK-9699] Test whether we should turn compression on or off by default for external sorting

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: SQL
• Target Version/s: 1.5.0
• Sprint: Spark 1.5 doc/QA sprint
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 06/Aug/15 18:55
• Updated: 11/Sep/15 07:33
• Resolved: 11/Sep/15 07:33

Description

As of Spark 1.4, compression is on by default for spilling, trading off CPU cycles for less I/O throughput usage. It is unclear whether it is better to have compression on or off when we spill data. We should test that and adjust the default setting accordingly.