[HIVE-7526] Research to use groupby transformation to replace Hive existing partitionByKey and SparkCollector combination

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Spark
• Assignee: Chao Sun
• Reporter: Xuefu Zhang
• Created: 27/Jul/14 20:17
• Updated: 29/May/15 02:28
• Resolved: 31/Jul/14 08:24

Description

Currently SparkClient shuffles data by calling paritionByKey(). This transformation outputs <key, value> tuples. However, Hive's ExecMapper expects <key, iterator<value>> tuples, and Spark's groupByKey() seems outputing this directly. Thus, using groupByKey, we may be able to avoid its own key clustering mechanism (in HiveReduceFunction). This research is to have a try.

Issue Links

• is part of: HIVE-7292 Hive on Spark
• is related to: HIVE-7493 Enhance HiveReduceFunction's row clustering [Spark Branch]