Hive
  1. Hive
  2. HIVE-5199

Custom SerDe containing a nonSettable complex data type row object inspector throws cast exception with HIVE 0.11

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      The issue happens because of the changes in HIVE-3833.

      Consider a partitioned table with different custom serdes for the partition and tables. The serde at table level, say, customSerDe1's object inspector is of settableDataType where as the serde at partition level, say, customSerDe2's object inspector is of nonSettableDataType. The current implementation introduced by HIVE-3833 does not convert nested Complex Data Types which extend nonSettableObjectInspector to a settableObjectInspector type inside ObjectInspectorConverters.getConvertedOI(). However, it tries to typecast the nonSettableObjectInspector to a settableObjectInspector inside ObjectInspectorConverters.getConverter(ObjectInspector inputOI, ObjectInspector outputOI).

      The attached patch HIVE-5199.2.patch.txt contains a stand-alone test case.

      The below exception can happen via FetchOperator as well as MapOperator.
      For example, consider the FetchOperator.
      Inside FetchOperator consider the following call:
      getRecordReader()->ObjectInspectorConverters. getConverter()

      The stack trace as follows:
      2013-08-28 17:57:25,307 ERROR CliDriver (SessionState.java:printError(432)) - Failed with exception java.io.IOException:java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
      java.io.IOException: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
      at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
      at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
      at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
      at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)
      at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
      at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
      at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
      at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
      at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
      Caused by: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
      at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:144)
      at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.<init>(ObjectInspectorConverters.java:307)
      at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:138)
      at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:406)
      at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)

      1. HIVE-5199-0.12-branch.conflict.patch
        0.9 kB
        Jason Dere
      2. 0001-HIVE-5199-0.12-branch.patch
        28 kB
        Hari Sankar Sivarama Subramaniyan
      3. HIVE-5199.patch.4.txt
        28 kB
        Hari Sankar Sivarama Subramaniyan
      4. HIVE-5199.3.patch.txt
        26 kB
        Hari Sankar Sivarama Subramaniyan
      5. HIVE-5199.2.patch.txt
        32 kB
        Hari Sankar Sivarama Subramaniyan
      6. HIVE-5199.patch.txt
        6 kB
        Hari Sankar Sivarama Subramaniyan

        Activity

        Hari Sankar Sivarama Subramaniyan created issue -
        Hari Sankar Sivarama Subramaniyan made changes -
        Field Original Value New Value
        Attachment HIVE-5199.patch.txt [ 12601447 ]
        Hari Sankar Sivarama Subramaniyan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hari Sankar Sivarama Subramaniyan made changes -
        Summary Read Only Custom SerDe works with HDP 1.1 but not with HDP 1.3 Custom SerDe containing a nonSettable complex data type row object inspector throws cast exception with HIVE 0.11
        Affects Version/s 0.11.0 [ 12323587 ]
        Description Custom serdes which used to work in HDP 1.1 is no longer working with HDP 1.3.
        The issue happens when the partition serde is not of settable type in HDP 1.3. The below exception happens via FetchOperator as well as MapOperator.

        Inside FetchOperator consider the following call:
        getRecordReader()->ObjectInspectorConverters. getConverter()
        The output object inspector is of settable type(because it is generated via ObjectInspectorConverters.getConvertedOI()) where as the input object inspector that gets passed as serde.getObjectorInspector() and is non-settable. Inside getConverter(), the (inputOI.equals(outputOI)) check fails and the switch statement tries to cast the non-settable object inspector to a settable object inspector.

        The stack trace as follows:
        2013-08-28 17:57:25,307 ERROR CliDriver (SessionState.java:printError(432)) - Failed with exception java.io.IOException:java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
        java.io.IOException: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
        at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
        at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
        Caused by: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:144)
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.<init>(ObjectInspectorConverters.java:307)
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:138)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:406)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
        The issue happens because of the changes in HIVE-3833.

        Consider a partitioned table with different custom serdes for the partition and tables. The serde at table level, say, customSerDe1's object inspector is of settableDataType where as the serde at partition level, say, customSerDe2's object inspector is of nonSettableDataType. The current implementation introduced by HIVE-3833 does not convert nested Complex Data Types which extend nonSettableObjectInspector to a settableObjectInspector type inside ObjectInspectorConverters.getConvertedOI(). However, it tries to typecast the nonSettableObjectInspector to a settableObjectInspector inside ObjectInspectorConverters.getConverter(ObjectInspector inputOI, ObjectInspector outputOI).

        The attached patch HIVE-5199.2.patch.txt contains a stand-alone test case.

        The below exception can happen via FetchOperator as well as MapOperator.
        For example, consider the FetchOperator.
        Inside FetchOperator consider the following call:
        getRecordReader()->ObjectInspectorConverters. getConverter()
        The output object inspector is of settable type(because it is generated via ObjectInspectorConverters.getConvertedOI()) where as the input object inspector that gets passed as serde.getObjectorInspector() and is non-settable. Inside getConverter(), the (inputOI.equals(outputOI)) check fails and the switch statement tries to cast the non-settable object inspector to a settable object inspector.

        The stack trace as follows:
        2013-08-28 17:57:25,307 ERROR CliDriver (SessionState.java:printError(432)) - Failed with exception java.io.IOException:java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
        java.io.IOException: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
        at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
        at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
        Caused by: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:144)
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.<init>(ObjectInspectorConverters.java:307)
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:138)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:406)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
        Hari Sankar Sivarama Subramaniyan made changes -
        Description The issue happens because of the changes in HIVE-3833.

        Consider a partitioned table with different custom serdes for the partition and tables. The serde at table level, say, customSerDe1's object inspector is of settableDataType where as the serde at partition level, say, customSerDe2's object inspector is of nonSettableDataType. The current implementation introduced by HIVE-3833 does not convert nested Complex Data Types which extend nonSettableObjectInspector to a settableObjectInspector type inside ObjectInspectorConverters.getConvertedOI(). However, it tries to typecast the nonSettableObjectInspector to a settableObjectInspector inside ObjectInspectorConverters.getConverter(ObjectInspector inputOI, ObjectInspector outputOI).

        The attached patch HIVE-5199.2.patch.txt contains a stand-alone test case.

        The below exception can happen via FetchOperator as well as MapOperator.
        For example, consider the FetchOperator.
        Inside FetchOperator consider the following call:
        getRecordReader()->ObjectInspectorConverters. getConverter()
        The output object inspector is of settable type(because it is generated via ObjectInspectorConverters.getConvertedOI()) where as the input object inspector that gets passed as serde.getObjectorInspector() and is non-settable. Inside getConverter(), the (inputOI.equals(outputOI)) check fails and the switch statement tries to cast the non-settable object inspector to a settable object inspector.

        The stack trace as follows:
        2013-08-28 17:57:25,307 ERROR CliDriver (SessionState.java:printError(432)) - Failed with exception java.io.IOException:java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
        java.io.IOException: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
        at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
        at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
        Caused by: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:144)
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.<init>(ObjectInspectorConverters.java:307)
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:138)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:406)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
        The issue happens because of the changes in HIVE-3833.

        Consider a partitioned table with different custom serdes for the partition and tables. The serde at table level, say, customSerDe1's object inspector is of settableDataType where as the serde at partition level, say, customSerDe2's object inspector is of nonSettableDataType. The current implementation introduced by HIVE-3833 does not convert nested Complex Data Types which extend nonSettableObjectInspector to a settableObjectInspector type inside ObjectInspectorConverters.getConvertedOI(). However, it tries to typecast the nonSettableObjectInspector to a settableObjectInspector inside ObjectInspectorConverters.getConverter(ObjectInspector inputOI, ObjectInspector outputOI).

        The attached patch HIVE-5199.2.patch.txt contains a stand-alone test case.

        The below exception can happen via FetchOperator as well as MapOperator.
        For example, consider the FetchOperator.
        Inside FetchOperator consider the following call:
        getRecordReader()->ObjectInspectorConverters. getConverter()

        The stack trace as follows:
        2013-08-28 17:57:25,307 ERROR CliDriver (SessionState.java:printError(432)) - Failed with exception java.io.IOException:java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
        java.io.IOException: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
        at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
        at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
        Caused by: java.lang.ClassCastException: com.skype.data.whaleshark.hadoop.hive.proto.ProtoMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.SettableMapObjectInspector
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:144)
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.<init>(ObjectInspectorConverters.java:307)
        at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:138)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:406)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
        Hari Sankar Sivarama Subramaniyan made changes -
        Attachment HIVE-5199.2.patch.txt [ 12601541 ]
        Attachment CustomSerDes.jar [ 12601542 ]
        Hari Sankar Sivarama Subramaniyan made changes -
        Attachment HIVE-5199.2.patch.txt [ 12601541 ]
        Hari Sankar Sivarama Subramaniyan made changes -
        Attachment HIVE-5199.2.patch.txt [ 12601543 ]
        Hari Sankar Sivarama Subramaniyan made changes -
        Attachment CustomSerDes.jar [ 12601542 ]
        Hari Sankar Sivarama Subramaniyan made changes -
        Attachment HIVE-5199.3.patch.txt [ 12601756 ]
        Hari Sankar Sivarama Subramaniyan made changes -
        Attachment HIVE-5199.patch.4.txt [ 12601951 ]
        Ashutosh Chauhan made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.13.0 [ 12324986 ]
        Resolution Fixed [ 1 ]
        Hari Sankar Sivarama Subramaniyan made changes -
        Attachment 0001-HIVE-5199-0.12-branch.patch [ 12603744 ]
        Thejas M Nair made changes -
        Fix Version/s 0.12.0 [ 12324312 ]
        Fix Version/s 0.13.0 [ 12324986 ]
        Jason Dere made changes -
        Attachment HIVE-5199-0.12-branch.conflict.patch [ 12605345 ]
        Ashutosh Chauhan made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Hari Sankar Sivarama Subramaniyan
            Reporter:
            Hari Sankar Sivarama Subramaniyan
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development