[SPARK-19525] Enable Compression of RDD Checkpoints

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• Component/s: Spark Core
• Assignee: Aaditya Ramesh
• Reporter: Aaditya Ramesh
• Shepherd: Shixiong Zhu
• Created: 09/Feb/17 00:55
• Updated: 28/Apr/17 22:30
• Resolved: 28/Apr/17 22:30

Description

In our testing, compressing partitions while writing them to checkpoints on HDFS using snappy helped performance significantly while also reducing the variability of the checkpointing operation. In our tests, checkpointing time was reduced by 3X, and variability was reduced by 2X for data sets of compressed size approximately 1 GB.

Issue Links

• links to: [Github] Pull Request #17024 (aramesh117)
• links to: [Github] Pull Request #17789 (zsxwing)