Description
How to reproduce this issue:
1. Add delta-core_2.12-1.0.0-SNAPSHOT.jar to ${SPARK_HOME}/jars. 2. bin/spark-shell --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
spark.sql("create table t1 using parquet as select id, id as lower_id from range(5)") spark.sql("CREATE VIEW v1 as SELECT * FROM t1") spark.sql("CREATE TABLE t2 USING DELTA PARTITIONED BY (LOWER_ID) SELECT LOWER_ID, ID FROM v1") spark.sql("CREATE TABLE t3 USING PARQUET PARTITIONED BY (LOWER_ID) SELECT LOWER_ID, ID FROM v1") spark.sql("desc extended t2").show(false) spark.sql("desc extended t3").show(false)
scala> spark.sql("desc extended t2").show(false) +----------------------------+--------------------------------------------------------------------------+-------+ |col_name |data_type |comment| +----------------------------+--------------------------------------------------------------------------+-------+ |lower_id |bigint | | |id |bigint | | | | | | |# Partitioning | | | |Part 0 |lower_id | | | | | | |# Detailed Table Information| | | |Name |default.t2 | | |Location |file:/Users/yumwang/Downloads/spark-3.1.1-bin-hadoop2.7/spark-warehouse/t2| | |Provider |delta | | |Table Properties |[Type=MANAGED,delta.minReaderVersion=1,delta.minWriterVersion=2] | | +----------------------------+--------------------------------------------------------------------------+-------+ scala> spark.sql("desc extended t3").show(false) +----------------------------+--------------------------------------------------------------------------+-------+ |col_name |data_type |comment| +----------------------------+--------------------------------------------------------------------------+-------+ |ID |bigint |null | |LOWER_ID |bigint |null | |# Partition Information | | | |# col_name |data_type |comment| |LOWER_ID |bigint |null | | | | | |# Detailed Table Information| | | |Database |default | | |Table |t3 | | |Owner |yumwang | | |Created Time |Mon Jul 12 14:07:16 CST 2021 | | |Last Access |UNKNOWN | | |Created By |Spark 3.1.1 | | |Type |MANAGED | | |Provider |PARQUET | | |Location |file:/Users/yumwang/Downloads/spark-3.1.1-bin-hadoop2.7/spark-warehouse/t3| | |Serde Library |org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe | | |InputFormat |org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat | | |OutputFormat |org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat | | |Partition Provider |Catalog | | +----------------------------+--------------------------------------------------------------------------+-------+