Hive
  1. Hive
  2. HIVE-4409

Prevent incompatible column type changes

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.12.0
    • Component/s: CLI, Metastore
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Added new config param hive.metastore.disallow.incompatible.col.type.changes which is false by default.

      Description

      If a user changes the type of an existing column of a partitioned table to an incompatible type, subsequent accesses of old partitions will result in a ClassCastException (see example below). We should prevent the user from making incompatible type changes. This feature will be controlled by a new config parameter.

      Example:

      CREATE TABLE test_table123 (a INT, b MAP<STRING, STRING>) PARTITIONED BY (ds STRING) STORED AS SEQUENCEFILE;
      INSERT OVERWRITE TABLE test_table123 PARTITION(ds="foo1") SELECT 1, MAP("a1", "b1") FROM src LIMIT 1;
      SELECT * from test_table123 WHERE ds="foo1";
      ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING);
      SELECT * from test_table123 WHERE ds="foo1";

      The last SELECT fails with the following exception:

      Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
      java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
      at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
      at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
      at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
      at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406)
      at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
      at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
      at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
      at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
      at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790)
      at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124)
      at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108)

      The ALTER TABLE statement is blocked if you set the following parameter, introduced int the fix to this JIRA:
      SET hive.metastore.disallow.incompatible.col.type.changes=true;

      1. hive.4409.1.patch
        22 kB
        Namit Jain
      2. HIVE-4409.D10539.1.patch
        20 kB
        Phabricator
      3. HIVE-4409.D10539.2.patch
        21 kB
        Phabricator

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Dilip Joseph
              Reporter:
              Dilip Joseph
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development