Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4644

Implement skewed join

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Won't Fix
    • None
    • None
    • Spark Core
    • None

    Description

      Skewed data is not rare. For example, a book recommendation site may have several books which are liked by most of the users. Running ALS on such skewed data will raise a OutOfMemory error, if some book has too many users which cannot be fit into memory. To solve it, we propose a skewed join implementation.

      Attachments

        1. Skewed Join Design Doc.pdf
          51 kB
          Shixiong Zhu

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-11387 minimize shuffles during joins by using existing partitions and bundling messages

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #3505 [Github] Pull Request #3505 (zsxwing)

          Pull request #6883 [Github] Pull Request #6883 (koertkuipers)

          Activity

            People

              Unassigned Unassigned
              zsxwing Shixiong Zhu
              Votes:
              6 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: