Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23021

AnalysisBarrier should not cut off the explain output for Parsed Logical Plan

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.3.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      In PR#20094 as a follow up to SPARK-20392, there were some fixes to the handling of AnalysisBarrier, but there seem to be more cases that need to be fixed.

      One such case is that right now the Parsed Logical Plan in explain output would be cutoff by AnalysisBarrier, e.g.

      scala> val df1 = spark.range(1).select('id as 'x, 'id + 1 as 'y).repartition(1).select('x === 'y)
      df1: org.apache.spark.sql.DataFrame = [(x = y): boolean]
      
      scala> df1.explain(true)
      == Parsed Logical Plan ==
      'Project [('x = 'y) AS (x = y)#22]
      +- AnalysisBarrier Repartition 1, true
      
      == Analyzed Logical Plan ==
      (x = y): boolean
      Project [(x#16L = y#17L) AS (x = y)#22]
      +- Repartition 1, true
         +- Project [id#13L AS x#16L, (id#13L + cast(1 as bigint)) AS y#17L]
            +- Range (0, 1, step=1, splits=Some(8))
      
      == Optimized Logical Plan ==
      Project [(x#16L = y#17L) AS (x = y)#22]
      +- Repartition 1, true
         +- Project [id#13L AS x#16L, (id#13L + 1) AS y#17L]
            +- Range (0, 1, step=1, splits=Some(8))
      
      == Physical Plan ==
      *Project [(x#16L = y#17L) AS (x = y)#22]
      +- Exchange RoundRobinPartitioning(1)
         +- *Project [id#13L AS x#16L, (id#13L + 1) AS y#17L]
            +- *Range (0, 1, step=1, splits=8)
      

        Attachments

          Activity

            People

            • Assignee:
              maropu Takeshi Yamamuro
              Reporter:
              rednaxelafx Kris Mok
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: