[SPARK-26435] Support creating partitioned table using Hive CTAS by specifying partition column names

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 24/Dec/18 11:47
• Updated: 27/Dec/18 08:07
• Resolved: 27/Dec/18 08:04

Description

Spark SQL doesn't support creating partitioned table using Hive CTAS in SQL syntax. However it is supported by using DataFrameWriter API.

val df = Seq(("a", 1)).toDF("part", "id")
df.write.format("hive").partitionBy("part").saveAsTable("t")

Hive begins to support this in newer version: https://issues.apache.org/jira/browse/HIVE-20241:

CREATE TABLE t PARTITIONED BY (part) AS SELECT 1 as id, "a" as part

To match DataFrameWriter API, we should this support to SQL syntax.

Issue Links

• links to: GitHub Pull Request #23376