[SPARK-21085] Failed to read the partitioned table created by Spark 2.1

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.0
• Component/s: SQL
• Target Version/s: 2.2.0
• Assignee: Xiao Li
• Reporter: Xiao Li
• Created: 13/Jun/17 23:25
• Updated: 14/Jun/17 08:28
• Resolved: 14/Jun/17 08:28

Description

Spark 2.2 is unable to read the partitioned table created by Spark 2.1 when the table schema does not put the partitioning column at the end of the schema.

assert(partitionFields.map(_.name) == partitionColumnNames)

The codes are from the following files:

https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala#L234-L236

When reading the table metadata from the metastore, we also need to reorder the columns.

Issue Links

• links to: [Github] Pull Request #18295 (gatorsmile)