Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6212

The EXPLAIN output of CTAS only shows the analyzed plan

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.3.0
    • 1.5.0
    • SQL
    • None
    • Spark 1.5 release

    Description

      When you try

      sql("explain extended create table parquet2 as select * from parquet1").collect.foreach(println)
      

      The output will be

      [== Parsed Logical Plan ==]
      ['CreateTableAsSelect None, parquet2, false, Some(TOK_CREATETABLE)]
      [ 'Project [*]]
      [  'UnresolvedRelation [parquet1], None]
      []
      [== Analyzed Logical Plan ==]
      [CreateTableAsSelect [Database:default, TableName: parquet2, InsertIntoHiveTable]]
      [Project [str#44]]
      [ Subquery parquet1]
      [  Relation[str#44] ParquetRelation2(List(file:/user/hive/warehouse/parquet1),Map(serialization.format -> 1, path -> file:/user/hive/warehouse/parquet1),Some(StructType(StructField(str,StringType,true))),None)]
      []
      []
      [== Optimized Logical Plan ==]
      [CreateTableAsSelect [Database:default, TableName: parquet2, InsertIntoHiveTable]]
      [Project [str#44]]
      [ Subquery parquet1]
      [  Relation[str#44] ParquetRelation2(List(file:/user/hive/warehouse/parquet1),Map(serialization.format -> 1, path -> file:/user/hive/warehouse/parquet1),Some(StructType(StructField(str,StringType,true))),None)]
      []
      []
      [== Physical Plan ==]
      [ExecutedCommand (CreateTableAsSelect [Database:default, TableName: parquet2, InsertIntoHiveTable]]
      [Project [str#44]]
      [ Subquery parquet1]
      [  Relation[str#44] ParquetRelation2(List(file:/user/hive/warehouse/parquet1),Map(serialization.format -> 1, path -> file:/user/hive/warehouse/parquet1),Some(StructType(StructField(str,StringType,true))),None)]
      [)]
      []
      [Code Generation: false]
      [== RDD ==]
      

      Query Plans of the SELECT clause shown in Optimized Plan and Physical Plan are actually analyzed plan.

      Attachments

        Activity

          People

            yijieshen Yijie Shen
            yhuai Yin Huai
            Yin Huai Yin Huai
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: