[SPARK-17612] Support `DESCRIBE table PARTITION` SQL syntax

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.2, 2.1.0
• Component/s: SQL
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 20/Sep/16 19:56
• Updated: 28/Oct/16 21:04
• Resolved: 29/Sep/16 22:33

Description

This issue implements `DESC PARTITION` SQL Syntax again. It was dropped since Spark 2.0.0.

Spark 2.0.0

scala> sql("CREATE TABLE partitioned_table (a STRING, b INT) PARTITIONED BY (c STRING, d STRING)")
res0: org.apache.spark.sql.DataFrame = []

scala> sql("ALTER TABLE partitioned_table ADD PARTITION (c='Us', d=1)")
res1: org.apache.spark.sql.DataFrame = []

scala> sql("DESC partitioned_table PARTITION (c='Us', d=1)").show(false)
org.apache.spark.sql.catalyst.parser.ParseException:
Unsupported SQL statement
== SQL ==
DESC partitioned_table PARTITION (c='Us', d=1)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:58)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:53)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:82)
  at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:45)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:53)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:573)
  ... 48 elided

Spark 1.6.2

scala> sql("CREATE TABLE partitioned_table (a STRING, b INT) PARTITIONED BY (c STRING, d STRING)")
res1: org.apache.spark.sql.DataFrame = [result: string]

scala> sql("ALTER TABLE partitioned_table ADD PARTITION (c='Us', d=1)")
res2: org.apache.spark.sql.DataFrame = [result: string]

scala> sql("DESC partitioned_table PARTITION (c='Us', d=1)").show(false)
16/09/20 12:48:36 WARN LazyStruct: Extra bytes detected at the end of the row! Ignoring similar problems.
+----------------------------------------------------------------+
|result                                                          |
+----------------------------------------------------------------+
|a                      string                                        |
|b                      int                                           |
|c                      string                                        |
|d                      string                                        |
|                                                                            |
|# Partition Information                                                      |
|# col_name             data_type               comment             |
|                                                                            |
|c                      string                                        |
|d                      string                                        |
+----------------------------------------------------------------+

Issue Links

• is duplicated by: SPARK-17452 Spark 2.0.0 is not supporting the "partition" keyword on a "describe" statement when using Hive Support
• links to: [Github] Pull Request #15168 (dongjoon-hyun)
• links to: [Github] Pull Request #15351 (dongjoon-hyun)