Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19620

Incorrect exchange coordinator Id in physical plan

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.1.0
    • 2.2.0
    • SQL
    • None

    Description

      When adaptive execution is enabled, an exchange coordinator is used to in the Exchange operators. For Join, the same exchange coordinator is used for its two Exchanges. But the physical plan shows two different coordinator Ids which is confusing.

      Here is an example:

      == Physical Plan ==
      *Project [key1#3L, value2#12L]
      +- *SortMergeJoin [key1#3L], [key2#11L], Inner
         :- *Sort [key1#3L ASC NULLS FIRST], false, 0
         :  +- Exchange(coordinator id: 1804587700) hashpartitioning(key1#3L, 10), coordinator[target post-shuffle partition size: 67108864]
         :     +- *Project [(id#0L % 500) AS key1#3L]
         :        +- *Filter isnotnull((id#0L % 500))
         :           +- *Range (0, 1000, step=1, splits=Some(10))
         +- *Sort [key2#11L ASC NULLS FIRST], false, 0
            +- Exchange(coordinator id: 793927319) hashpartitioning(key2#11L, 10), coordinator[target post-shuffle partition size: 67108864]
               +- *Project [(id#8L % 500) AS key2#11L, id#8L AS value2#12L]
                  +- *Filter isnotnull((id#8L % 500))
                     +- *Range (0, 1000, step=1, splits=Some(10))
      
      

      Attachments

        Activity

          People

            carsonwang Carson Wang
            carsonwang Carson Wang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: