Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-5199

Custom SerDe containing a nonSettable complex data type row object inspector throws cast exception with HIVE 0.11

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      The issue happens because of the changes in HIVE-3833.

      Consider a partitioned table with different custom serdes for the partition and tables. The serde at table level, say, customSerDe1's object inspector is of settableDataType where as the serde at partition level, say, customSerDe2's object inspector is of nonSettableDataType. The current implementation introduced by HIVE-3833 does not convert nested Complex Data Types which extend nonSettableObjectInspector to a settableObjectInspector type inside ObjectInspectorConverters.getConvertedOI(). However, it tries to typecast the nonSettableObjectInspector to a settableObjectInspector inside ObjectInspectorConverters.getConverter(ObjectInspector inputOI, ObjectInspector outputOI).

      The attached patch HIVE-5199.2.patch.txt contains a stand-alone test case.

      The below exception can happen via FetchOperator as well as MapOperator.
      For example, consider the FetchOperator.
      Inside FetchOperator consider the following call:
      getRecordReader()->ObjectInspectorConverters. getConverter()

      The stack trace as follows:
      2013-08-28 17:57:25,307 ERROR CliDriver (SessionState.java:printError(432)) - Failed with exception java.io.IOException:java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
      java.io.IOException: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
      at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
      at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
      at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
      at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)
      at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
      at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
      at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
      at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
      at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
      Caused by: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
      at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:144)
      at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.<init>(ObjectInspectorConverters.java:307)
      at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:138)
      at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:406)
      at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)

        Attachments

        1. HIVE-5199-0.12-branch.conflict.patch
          0.9 kB
          Jason Dere
        2. HIVE-5199.patch.txt
          6 kB
          Hari Sankar Sivarama Subramaniyan
        3. HIVE-5199.patch.4.txt
          28 kB
          Hari Sankar Sivarama Subramaniyan
        4. HIVE-5199.3.patch.txt
          26 kB
          Hari Sankar Sivarama Subramaniyan
        5. HIVE-5199.2.patch.txt
          32 kB
          Hari Sankar Sivarama Subramaniyan
        6. 0001-HIVE-5199-0.12-branch.patch
          28 kB
          Hari Sankar Sivarama Subramaniyan

          Activity

            People

            • Assignee:
              hsubramaniyan Hari Sankar Sivarama Subramaniyan
              Reporter:
              hsubramaniyan Hari Sankar Sivarama Subramaniyan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: