[SPARK-2978] Provide an MR-style shuffle transformation

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Spark Core
• Target Version/s: 1.2.0
• Assignee: Sandy Ryza
• Reporter: Sandy Ryza
• Created: 12/Aug/14 00:57
• Updated: 08/Sep/14 18:21
• Resolved: 08/Sep/14 18:21

Description

For Hive on Spark joins in particular, and for running legacy MR code in general, I think it would be useful to provide a transformation with the semantics of the Hadoop MR shuffle, i.e. one that

• groups by key: provides (Key, Iterator[Value])
• within each partition, provides keys in sorted order

A couple ways that could make sense to expose this:

• Add a new operator. "groupAndSortByKey", "groupByKeyAndSortWithinPartition", "hadoopStyleShuffle", maybe?
• Allow groupByKey to take an ordering param for keys within a partition

Issue Links

• is depended upon by: HIVE-7384 Research into reduce-side join [Spark Branch]
• is depended upon by: SPARK-3145 Hive on Spark umbrella
• is related to: SPARK-2926 Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle
• links to: [Github] Pull Request #2274 (sryza)