Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-9850 Adaptive execution in Spark
  3. SPARK-9862

Join: Handling data skew

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      For a two way shuffle join, if one or multiple groups are skewed in one table (say left table) but having a relative small number of rows in another table (say right table), we can use broadcast join for these skewed groups and use shuffle join for other groups.

        Attachments

          Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-16753 Spark SQL doesn't handle skewed dataset joins properly

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-17788 RangePartitioner results in few very large tasks and many small to empty tasks

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SPARK-24799 A solution of dealing with data skew in left,right,inner join

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #15297 [Github] Pull Request #15297 (YuhuWang2002)

          Web Link GitHub Pull Request #15297

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                yhuai Yin Huai
              • Votes:
                9 Vote for this issue
                Watchers:
                40 Start watching this issue

                Dates

                • Created:
                  Updated: