[SPARK-17166] CTAS lost table properties after conversion to data source tables.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.4.0
• Component/s: SQL
• Assignee: Dongjoon Hyun
• Reporter: Xiao Li
• Created: 20/Aug/16 05:46
• Updated: 07/Nov/18 00:36
• Resolved: 07/Nov/18 00:36

Description

CTAS lost table properties after conversion to data source tables. For example,

CREATE TABLE t TBLPROPERTIES('prop1' = 'c', 'prop2' = 'd') AS SELECT 1 as a, 1 as b

The output of `DESC FORMATTED t` does not have the related properties.

|Table Parameters:           |                                                                                                              |       |
|  rawDataSize               |-1                                                                                                            |       |
|  numFiles                  |1                                                                                                             |       |
|  transient_lastDdlTime     |1471670983                                                                                                    |       |
|  totalSize                 |496                                                                                                           |       |
|  spark.sql.sources.provider|parquet                                                                                                       |       |
|  EXTERNAL                  |FALSE                                                                                                         |       |
|  COLUMN_STATS_ACCURATE     |false                                                                                                         |       |
|  numRows                   |-1                                                                                                            |       |
|                            |                                                                                                              |       |
|# Storage Information       |                                                                                                              |       |
|SerDe Library:              |org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe                                                   |       |
|InputFormat:                |org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat                                                 |       |
|OutputFormat:               |org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat                                                |       |
|Compressed:                 |No                                                                                                            |       |
|Storage Desc Parameters:    |                                                                                                              |       |
|  serialization.format      |1                                                                                                             |       |
|  path                      |file:/private/var/folders/4b/sgmfldk15js406vk7lw5llzw0000gn/T/warehouse-f3aa2927-6464-4a35-a715-1300dde6c614/t|       |

Issue Links

• is superceded by: SPARK-22158 convertMetastore should not ignore storage properties
• is superceded by: SPARK-23355 convertMetastore should not ignore table properties
• links to: [Github] Pull Request #14727 (gatorsmile)