[SPARK-24799] A solution of dealing with data skew in left,right,inner join

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.0, 2.1.0, 2.2.0, 2.3.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: marymwu
• Created: 13/Jul/18 09:10
• Updated: 31/Aug/19 06:37
• Resolved: 15/Jul/18 05:17

Description

For the left,right,inner join statment execution, this solution is mainling about to devide the partions where the data skew has occured into serveral partions with smaller data scale, in order to parallelly execute more tasks to increase effeciency.

Issue Links

• duplicates: SPARK-9862 Join: Handling data skew
• links to: [Github] Pull Request #21783 (marymwu)
• links to: GitHub Pull Request #21783