[HIVE-8406] Research on skewed join [Spark Branch]

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Done
• Component/s: Spark
• Assignee: Rui Li
• Reporter: Xuefu Zhang
• Created: 08/Oct/14 21:00
• Updated: 17/Dec/14 16:35
• Resolved: 17/Dec/14 03:23

Description

Research on how to handle skewed join for hive on spark. Here is original hive's design doc for skewed join, https://cwiki.apache.org/confluence/display/Hive/Skewed+Join+Optimization.

Issue Links

• depends upon: HIVE-7613 Research optimization of auto convert join to map join [Spark branch]
• incorporates: HIVE-8958 Make sure map join tasks created by runtime skew join can fit into memory [Spark Branch]
• incorporates: HIVE-8535 Enable compile time skew join optimization for spark [Spark Branch]
• incorporates: HIVE-8536 Enable SkewJoinResolver for spark [Spark Branch]
• incorporates: HIVE-8913 Make SparkMapJoinResolver handle runtime skew join [Spark Branch]
• incorporates: HIVE-9097 Support runtime skew join for more queries [Spark Branch]