[SPARK-4550] In sort-based shuffle, store map outputs in serialized form

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.4.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 1.4.0
• Assignee: Sandy Ryza
• Reporter: Sandy Ryza
• Created: 22/Nov/14 01:06
• Updated: 05/May/15 19:59
• Resolved: 01/May/15 20:55

Description

One drawback with sort-based shuffle compared to hash-based shuffle is that it ends up storing many more java objects in memory. If Spark could store map outputs in serialized form, it could

• spill less often because the serialized form is more compact
• reduce GC pressure

This will only work when the serialized representations of objects are independent from each other and occupy contiguous segments of memory. E.g. when Kryo reference tracking is left on, objects may contain pointers to objects farther back in the stream, which means that the sort can't relocate objects without corrupting them.

Issue Links

• blocks: SPARK-6026 Eliminate the bypassMergeThreshold parameter and associated hash-ish shuffle within the Sort shuffle code
• is duplicated by: SPARK-2114 groupByKey and joins on raw data
• links to: [Github] Pull Request #4450 (sryza)
• links to: [Github] Pull Request #5916 (sryza)
• links to: Related discussion on Kryo group