[SPARK-22158] convertMetastore should not ignore storage properties

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 2.1.0, 2.2.0
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: SQL
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 28/Sep/17 17:23
• Updated: 09/Jul/19 22:45
• Resolved: 03/Oct/17 18:43

Description

From the beginning, convertMetastoreOrc ignores table properties and use an emtpy map instead. It's the same with convertMetastoreParquet.

val options = Map[String, String]()

• SPARK-14070: https://github.com/apache/spark/pull/11891/files#diff-ee66e11b56c21364760a5ed2b783f863R650
• master: https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala#L197

Issue Links

• blocks: SPARK-20901 Feature parity for ORC with Parquet
• relates to: SPARK-22926 Respect table-level conf compression codec `Compression` in multiple scenarios
• relates to: SPARK-23355 convertMetastore should not ignore table properties
• supercedes: SPARK-17166 CTAS lost table properties after conversion to data source tables.
• links to: [Github] Pull Request #19382 (dongjoon-hyun)
• links to: [Github] Pull Request #19417 (dongjoon-hyun)
• links to: [Github] Pull Request #20522 (dongjoon-hyun)