Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17166

CTAS lost table properties after conversion to data source tables.

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.4.0
    • Component/s: SQL
    • Labels:
      None

      Description

      CTAS lost table properties after conversion to data source tables. For example,

      CREATE TABLE t TBLPROPERTIES('prop1' = 'c', 'prop2' = 'd') AS SELECT 1 as a, 1 as b
      

      The output of `DESC FORMATTED t` does not have the related properties.

      |Table Parameters:           |                                                                                                              |       |
      |  rawDataSize               |-1                                                                                                            |       |
      |  numFiles                  |1                                                                                                             |       |
      |  transient_lastDdlTime     |1471670983                                                                                                    |       |
      |  totalSize                 |496                                                                                                           |       |
      |  spark.sql.sources.provider|parquet                                                                                                       |       |
      |  EXTERNAL                  |FALSE                                                                                                         |       |
      |  COLUMN_STATS_ACCURATE     |false                                                                                                         |       |
      |  numRows                   |-1                                                                                                            |       |
      |                            |                                                                                                              |       |
      |# Storage Information       |                                                                                                              |       |
      |SerDe Library:              |org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe                                                   |       |
      |InputFormat:                |org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat                                                 |       |
      |OutputFormat:               |org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat                                                |       |
      |Compressed:                 |No                                                                                                            |       |
      |Storage Desc Parameters:    |                                                                                                              |       |
      |  serialization.format      |1                                                                                                             |       |
      |  path                      |file:/private/var/folders/4b/sgmfldk15js406vk7lw5llzw0000gn/T/warehouse-f3aa2927-6464-4a35-a715-1300dde6c614/t|       |
      

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              dongjoon Dongjoon Hyun
              Reporter:
              smilegator Xiao Li

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment