[SPARK-6905] Upgrade Snappy Java to 1.1.1.7 to fix bug that resulted in worse compression

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.2, 1.3.1
• Fix Version/s: 1.2.3, 1.3.2, 1.4.0
• Component/s: Spark Core
• Target Version/s: 1.2.3, 1.3.2
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 14/Apr/15 17:34
• Updated: 21/Apr/15 06:36
• Resolved: 14/Apr/15 20:44

Description

We should upgrade our snappy-java dependency to 1.1.1.7 in order to include a fix for a bug that resulted in worse compression (see https://github.com/xerial/snappy-java/issues/100). I believe that this may partially fix SPARK-5081, an issue where the size of shuffle data increased from Spark 1.1.x to 1.2.0.

Issue Links

• is duplicated by: SPARK-5081 Shuffle write increases
• links to: [Github] Pull Request #5512 (JoshRosen)