Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-9850 Adaptive execution in Spark
  3. SPARK-11471

Improve the way that we plan shuffled join

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • None
    • None
    • SQL

    Description

      Right now, when adaptive query execution is enabled, in most of cases, we will shuffle input tables for every join. However, once we finish our work of https://issues.apache.org/jira/browse/SPARK-10665, we will be able to have a global on the input datasets of a stage. Then, we should be able to add exchange coordinators after we get the entire physical plan (after the phase that we add Exchanges).

      I will try to fill in more information later.

      Attachments

        Issue Links

          is blocked by

          Sub-task - The sub-task of the issue SPARK-10665 Connect the local iterators with the planner

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              yhuai Yin Huai
              Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: