Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17556

Executor side broadcast for broadcast joins

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Spark Core, SQL
    • Labels:
      None

      Description

      Currently in Spark SQL, in order to perform a broadcast join, the driver must collect the result of an RDD and then broadcast it. This introduces some extra latency. It might be possible to broadcast directly from executors.

        Attachments

        1. executor-side-broadcast.pdf
          67 kB
          L. C. Hsieh
        2. executor broadcast.pdf
          198 kB
          Fei Wang

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rxin Reynold Xin
            • Votes:
              9 Vote for this issue
              Watchers:
              35 Start watching this issue

              Dates

              • Created:
                Updated: