Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-9850 Adaptive execution in Spark
  3. SPARK-9862

Join: Handling data skew

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • None
    • None
    • SQL

    Description

      For a two way shuffle join, if one or multiple groups are skewed in one table (say left table) but having a relative small number of rows in another table (say right table), we can use broadcast join for these skewed groups and use shuffle join for other groups.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-16753 Spark SQL doesn't handle skewed dataset joins properly

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-17788 RangePartitioner results in few very large tasks and many small to empty tasks

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SPARK-24799 A solution of dealing with data skew in left,right,inner join

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #15297 [Github] Pull Request #15297 (YuhuWang2002)

          Web Link GitHub Pull Request #15297

          Activity

            People

              Unassigned Unassigned
              yhuai Yin Huai
              Votes:
              10 Vote for this issue
              Watchers:
              41 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: