Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15995

Syncing metastore table with serde schema

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.1, 2.1.0, 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: Metastore
    • Labels:
      None
    • Target Version/s:

      Description

      Hive enables table schema evolution via properties. For avro e.g. we can alter the 'avro.schema.url' property to update table schema to the next version. Updating properties however doesn't affect column list stored in metastore DB so the table is not in the newest version when returned from metastore API. This is problem for tools working with metastore (e.g. Presto).

      To solve this issue I suggest to introduce new DDL statement syncing metastore columns with those from serde:

      ALTER TABLE user_test1 UPDATE COLUMNS
      

      Note that this is format independent solution.

      To reproduce, follow the instructions below:

      • Create table based on avro schema version 1 (cxv1.avsc)
        CREATE EXTERNAL TABLE user_test1
          PARTITIONED BY (dt string)
          ROW FORMAT SERDE
          'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
          STORED AS INPUTFORMAT
          'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
          OUTPUTFORMAT
          'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
          LOCATION
          '/tmp/schema-evolution/user_test1'
          TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
        
      • Update schema to version 2 (cx2.avsc)
        ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = '/tmp/schema-evolution/cx2.avsc');
        
      • Print serde columns (top info) and metastore columns (Detailed Table Information):
        DESCRIBE EXTENDED user_test1
        

        Attachments

        1. cx1.avsc
          0.4 kB
          Michal Ferlinski
        2. cx2.avsc
          0.5 kB
          Michal Ferlinski
        3. HIVE-15995.1.patch
          26 kB
          Adam Szita
        4. HIVE-15995.2.patch
          26 kB
          Adam Szita
        5. HIVE-15995.3.patch
          26 kB
          Adam Szita
        6. HIVE-15995.4.patch
          27 kB
          Adam Szita
        7. HIVE-15995.5.patch
          46 kB
          Adam Szita
        8. HIVE-15995.patch
          9 kB
          Michal Ferlinski

          Activity

            People

            • Assignee:
              szita Adam Szita
              Reporter:
              michal.ferlinski Michal Ferlinski
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: