[SPARK-9862] Join: Handling data skew

• Type: Sub-task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Yin Huai
• Created: 12/Aug/15 04:39
• Updated: 15/Jul/18 05:17

Description

For a two way shuffle join, if one or multiple groups are skewed in one table (say left table) but having a relative small number of rows in another table (say right table), we can use broadcast join for these skewed groups and use shuffle join for other groups.

Issue Links

• is duplicated by: SPARK-16753 Spark SQL doesn't handle skewed dataset joins properly
• is duplicated by: SPARK-17788 RangePartitioner results in few very large tasks and many small to empty tasks
• is duplicated by: SPARK-24799 A solution of dealing with data skew in left,right,inner join
• links to: [Github] Pull Request #15297 (YuhuWang2002)