Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-10697

ObjectInspectorConvertors#UnionConvertor does a faulty conversion

    Details

      Description

      Currently the UnionConvertor in the ObjectInspectorConvertors class has an issue with the convert method where it attempts to convert the objectinspector itself instead of converting the field.[1]. This should be changed to convert the field itself. This could result in a ClassCastException as shown below:

      Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyUnionObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.lazy.LazyString
      	at org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyStringObjectInspector.getPrimitiveWritableObject(LazyStringObjectInspector.java:51)
      	at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter.convert(PrimitiveObjectInspectorConverter.java:391)
      	at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter.convert(PrimitiveObjectInspectorConverter.java:338)
      	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$UnionConverter.convert(ObjectInspectorConverters.java:456)
      	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:395)
      	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$MapConverter.convert(ObjectInspectorConverters.java:539)
      	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:395)
      	at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.readRow(MapOperator.java:154)
      	at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.access$200(MapOperator.java:127)
      	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:518)
      	... 9 more
      

      [1] https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java#L466

      1. HIVE-10697.2.patch.txt
        10 kB
        Swarnim Kulkarni
      2. HIVE-10697.1.patch.txt
        8 kB
        Swarnim Kulkarni

        Issue Links

          Activity

          Hide
          swarnim Swarnim Kulkarni added a comment -

          Hari Sankar Sivarama Subramaniyan Seems like this UnionConvertor was added as a part of HIVE-5202. Could you please help me understand why are we doing a conversion from one objectinspector to another in this code?[1] Aren't the ObjectinspectorConvertors supposed to be used to convert from one object to another given the two OIs and not the OIs themselves?

          [1] https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java#L466

          Show
          swarnim Swarnim Kulkarni added a comment - Hari Sankar Sivarama Subramaniyan Seems like this UnionConvertor was added as a part of HIVE-5202 . Could you please help me understand why are we doing a conversion from one objectinspector to another in this code? [1] Aren't the ObjectinspectorConvertors supposed to be used to convert from one object to another given the two OIs and not the OIs themselves? [1] https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java#L466
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Also seems like we have unit tests missing for the UnionConverter as well that should be added as a part of this JIRA.

          Show
          swarnim Swarnim Kulkarni added a comment - Also seems like we have unit tests missing for the UnionConverter as well that should be added as a part of this JIRA.
          Hide
          hsubramaniyan Hari Sankar Sivarama Subramaniyan added a comment -

          Swarnim Kulkarni You are right regarding the fact that ObjectinspectorConvertors are supposed to be used to convert from one object to another given the two OIs and not the OIs themselves. The below line needs to be modified to change fieldConverters.get(f).convert(inputOI) to fieldConverters.get(f).convert(input) :

              @Override
              public Object convert(Object input) {
          .....
                for (int f = 0; f < minFields; f++) {
                  Object outputFieldValue = fieldConverters.get(f).convert(inputOI);
           ....
              }
          

          Thanks
          Hari

          Show
          hsubramaniyan Hari Sankar Sivarama Subramaniyan added a comment - Swarnim Kulkarni You are right regarding the fact that ObjectinspectorConvertors are supposed to be used to convert from one object to another given the two OIs and not the OIs themselves. The below line needs to be modified to change fieldConverters.get(f).convert(inputOI) to fieldConverters.get(f).convert(input) : @Override public Object convert( Object input) { ..... for ( int f = 0; f < minFields; f++) { Object outputFieldValue = fieldConverters.get(f).convert(inputOI); .... } Thanks Hari
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Thanks Hari Sankar Sivarama Subramaniyan. Looks like I am still running into this after that:

          java.lang.ClassCastException: org.apache.hadoop.hive.serde2.objectinspector.StandardUnionObjectInspector$StandardUnion cannot be cast to java.lang.Integer
          	at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaIntObjectInspector.get(JavaIntObjectInspector.java:40)
          	at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getString(PrimitiveObjectInspectorUtils.java:810)
          	at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$StringConverter.convert(PrimitiveObjectInspectorConverter.java:446)
          	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$UnionConverter.convert(ObjectInspectorConverters.java:484)
          	at org.apache.hadoop.hive.serde2.objectinspector.TestObjectInspectorConverters.testObjectInspectorConverters(TestObjectInspectorConverters.java:201)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at junit.framework.TestCase.runTest(TestCase.java:176)
          	at junit.framework.TestCase.runBare(TestCase.java:141)
          	at junit.framework.TestResult$1.protect(TestResult.java:122)
          	at junit.framework.TestResult.runProtected(TestResult.java:142)
          	at junit.framework.TestResult.run(TestResult.java:125)
          	at junit.framework.TestCase.run(TestCase.java:129)
          	at junit.framework.TestSuite.runTest(TestSuite.java:255)
          	at junit.framework.TestSuite.run(TestSuite.java:250)
          	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
          	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
          	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
          	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
          	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
          	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
          	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
          
          Show
          swarnim Swarnim Kulkarni added a comment - Thanks Hari Sankar Sivarama Subramaniyan . Looks like I am still running into this after that: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.objectinspector.StandardUnionObjectInspector$StandardUnion cannot be cast to java.lang.Integer at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaIntObjectInspector.get(JavaIntObjectInspector.java:40) at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getString(PrimitiveObjectInspectorUtils.java:810) at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$StringConverter.convert(PrimitiveObjectInspectorConverter.java:446) at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$UnionConverter.convert(ObjectInspectorConverters.java:484) at org.apache.hadoop.hive.serde2.objectinspector.TestObjectInspectorConverters.testObjectInspectorConverters(TestObjectInspectorConverters.java:201) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:176) at junit.framework.TestCase.runBare(TestCase.java:141) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:255) at junit.framework.TestSuite.run(TestSuite.java:250) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
          Hide
          hsubramaniyan Hari Sankar Sivarama Subramaniyan added a comment -

          Swarnim Kulkarni Thanks for getting back on this, can you please provide the test case you are trying. It looks like the current code is faulty as you pointed out initially and will require modification.

          Thanks
          Hari

          Show
          hsubramaniyan Hari Sankar Sivarama Subramaniyan added a comment - Swarnim Kulkarni Thanks for getting back on this, can you please provide the test case you are trying. It looks like the current code is faulty as you pointed out initially and will require modification. Thanks Hari
          Hide
          hjarraya Haithem Jarraya added a comment -

          Hi,

          We are facing similar issue, is there a workaround?

          Thanks,

          Haithem

          Show
          hjarraya Haithem Jarraya added a comment - Hi, We are facing similar issue, is there a workaround? Thanks, Haithem
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Patch attached.

          Show
          swarnim Swarnim Kulkarni added a comment - Patch attached.
          Show
          swarnim Swarnim Kulkarni added a comment - RB: https://reviews.apache.org/r/37563/
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Hari Sankar Sivarama Subramaniyan Would you mind reviewing the patch?

          Show
          swarnim Swarnim Kulkarni added a comment - Hari Sankar Sivarama Subramaniyan Would you mind reviewing the patch?
          Hide
          hiveqa Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12750955/HIVE-10697.1.patch.txt

          SUCCESS: +1 9370 tests passed

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4993/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4993/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4993/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12750955 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12750955/HIVE-10697.1.patch.txt SUCCESS: +1 9370 tests passed Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4993/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4993/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4993/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12750955 - PreCommit-HIVE-TRUNK-Build
          Hide
          hsubramaniyan Hari Sankar Sivarama Subramaniyan added a comment -

          Swarnim Kulkarni Thanks for the patch, the patch looks good overall. There a few comments I have added in the review. Can you please take a look at it.

          Thanks
          Hari

          Show
          hsubramaniyan Hari Sankar Sivarama Subramaniyan added a comment - Swarnim Kulkarni Thanks for the patch, the patch looks good overall. There a few comments I have added in the review. Can you please take a look at it. Thanks Hari
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Thanks for the review Hari Sankar Sivarama Subramaniyan.

          Addressed the comments and updated the patch.

          Show
          swarnim Swarnim Kulkarni added a comment - Thanks for the review Hari Sankar Sivarama Subramaniyan . Addressed the comments and updated the patch.
          Hide
          swarnim Swarnim Kulkarni added a comment -

          The RB has been updated too.

          Show
          swarnim Swarnim Kulkarni added a comment - The RB has been updated too.
          Hide
          hsubramaniyan Hari Sankar Sivarama Subramaniyan added a comment -

          +1 pending unit test runs on patch#2.

          Thanks
          Hari

          Show
          hsubramaniyan Hari Sankar Sivarama Subramaniyan added a comment - +1 pending unit test runs on patch#2. Thanks Hari
          Hide
          hiveqa Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12751109/HIVE-10697.2.patch.txt

          SUCCESS: +1 9370 tests passed

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5007/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5007/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5007/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12751109 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12751109/HIVE-10697.2.patch.txt SUCCESS: +1 9370 tests passed Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5007/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5007/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5007/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12751109 - PreCommit-HIVE-TRUNK-Build
          Hide
          hsubramaniyan Hari Sankar Sivarama Subramaniyan added a comment -

          Committed to master, branch-1. Thanks Swarnim Kulkarni for the fix.

          Show
          hsubramaniyan Hari Sankar Sivarama Subramaniyan added a comment - Committed to master, branch-1. Thanks Swarnim Kulkarni for the fix.

            People

            • Assignee:
              swarnim Swarnim Kulkarni
              Reporter:
              swarnim Swarnim Kulkarni
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development