Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17556

Executor side broadcast for broadcast joins

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Spark Core, SQL
    • None

    Description

      Currently in Spark SQL, in order to perform a broadcast join, the driver must collect the result of an RDD and then broadcast it. This introduces some extra latency. It might be possible to broadcast directly from executors.

      Attachments

        1. executor broadcast.pdf
          198 kB
          Fei Wang
        2. executor-side-broadcast.pdf
          67 kB
          L. C. Hsieh

        Activity

          People

            Unassigned Unassigned
            rxin Reynold Xin
            Votes:
            10 Vote for this issue
            Watchers:
            40 Start watching this issue

            Dates

              Created:
              Updated: