Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18854

getNodeNumbered and generateTreeString are not consistent

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.3, 2.1.0
    • Component/s: SQL
    • Labels:
      None

      Description

      This is a bug introduced by subquery handling. generateTreeString numbers trees including innerChildren (used to print subqueries), but getNodeNumbered ignores that. As a result, getNodeNumbered is not always correct.

      Repro:

          val df = sql("select * from range(10) where id not in " +
            "(select id from range(2) union all select id from range(2))")
      
          println("-------------------------------------------------------")
          println(df.queryExecution.analyzed.numberedTreeString)
          println("-------------------------------------------------------")
      
          println("-------------------------------------------------------")
          println(df.queryExecution.analyzed(3))
          println("-------------------------------------------------------")
      

      Output looks like

      -------------------------------------------------------
      00 Project [id#1L]
      01 +- Filter NOT predicate-subquery#0 [(id#1L = id#2L)]
      02    :  +- Union
      03    :     :- Project [id#2L]
      04    :     :  +- Range (0, 2, step=1, splits=None)
      05    :     +- Project [id#3L]
      06    :        +- Range (0, 2, step=1, splits=None)
      07    +- Range (0, 10, step=1, splits=None)
      -------------------------------------------------------
      -------------------------------------------------------
      null
      -------------------------------------------------------
      

      Note that 3 should be the Project node, but getNodeNumbered ignores innerChild and as a result returns the wrong one.

        Attachments

          Activity

            People

            • Assignee:
              rxin Reynold Xin
              Reporter:
              rxin Reynold Xin
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: