[SPARK-23355] convertMetastore should not ignore table properties

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.2, 2.1.2, 2.2.1, 2.3.0
• Fix Version/s: 2.4.0
• Component/s: SQL
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 07/Feb/18 20:58
• Updated: 07/Nov/18 00:32
• Resolved: 27/Apr/18 03:02

Description

`convertMetastoreOrc/Parquet` ignores table properties.

This happens for a table created by `STORED AS ORC/PARQUET`.

Currently, `convertMetastoreParquet` is `true` by default. So, it's a well-known user-facing issue. For `convertMetastoreOrc`, it has been `false` and SPARK-22279 tried to turn on that for Feature Parity with Parquet.

However, it's indeed a regression for the existing Hive ORC table users. So, it'll be reverted for Apache Spark 2.3 via https://github.com/apache/spark/pull/20536 .

Issue Links

• blocks: SPARK-20901 Feature parity for ORC with Parquet
• is related to: SPARK-22158 convertMetastore should not ignore storage properties
• supercedes: SPARK-17166 CTAS lost table properties after conversion to data source tables.
• links to: [Github] Pull Request #20522 (dongjoon-hyun)
• links to: [Github] Pull Request #21269 (dongjoon-hyun)