[SPARK-13926] Automatically use Kryo serializer when shuffling RDDs with simple types

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 2.0.0
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 16/Mar/16 06:22
• Updated: 17/May/20 18:31
• Resolved: 17/Mar/16 05:52

Description

Because ClassTags are available when constructing ShuffledRDD we can use them to automatically use Kryo for shuffle serialization when the RDD's types are guaranteed to be compatible with Kryo (e.g. RDDs whose key, value, and/or combiner types are primitives, arrays of primitives, or strings). This is likely to result in a large performance gain for many RDD API workloads.

Issue Links

• links to: [Github] Pull Request #11755 (JoshRosen)