Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Query Processor
    • Labels:

      Description

      hive> CREATE TABLE DEST1(key UNIONTYPE<STRING, STRING>, value BIGINT) STORED AS TEXTFILE;
      OK
      Time taken: 0.076 seconds
      hive> CREATE TABLE DEST2(key UNIONTYPE<STRING, STRING>, value BIGINT) STORED AS TEXTFILE;
      OK
      Time taken: 0.034 seconds
      hive> SELECT * FROM DEST1 JOIN DEST2 on (DEST1.key = DEST2.key);
      

        Issue Links

          Activity

          Ashutosh Chauhan created issue -
          Hide
          Ashutosh Chauhan added a comment -

          NPE is thrown in setup task of Reducer:

          java.lang.RuntimeException: Error in configuring object
          	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
          	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
          	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
          	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:484)
          	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:419)
          	at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Subject.java:396)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
          	at org.apache.hadoop.mapred.Child.main(Child.java:253)
          Caused by: java.lang.reflect.InvocationTargetException
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
          	... 9 more
          Caused by: java.lang.RuntimeException: Reduce operator initialization failed
          	at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:157)
          	... 14 more
          Caused by: java.lang.NullPointerException
          	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardObjectInspector(ObjectInspectorUtils.java:105)
          	at org.apache.hadoop.hive.ql.exec.JoinUtil.getStandardObjectInspectors(JoinUtil.java:97)
          	at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.initializeOp(CommonJoinOperator.java:255)
          	at org.apache.hadoop.hive.ql.exec.JoinOperator.initializeOp(JoinOperator.java:59)
          	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
          	at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:150)
          	... 14 more
          
          Show
          Ashutosh Chauhan added a comment - NPE is thrown in setup task of Reducer: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:484) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:419) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) ... 9 more Caused by: java.lang.RuntimeException: Reduce operator initialization failed at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:157) ... 14 more Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardObjectInspector(ObjectInspectorUtils.java:105) at org.apache.hadoop.hive.ql.exec.JoinUtil.getStandardObjectInspectors(JoinUtil.java:97) at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.initializeOp(CommonJoinOperator.java:255) at org.apache.hadoop.hive.ql.exec.JoinOperator.initializeOp(JoinOperator.java:59) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357) at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:150) ... 14 more
          Hide
          Ashutosh Chauhan added a comment -

          Thinking more about this. I dont think we want to support join on union type, since in general real data in a union could be of any type which makes it impossible to do comparison of union types correctly. It can be argued that in some special cases that could be achieved via type widening, but I would argue selecting union type for storing such data which later need to be joined is a wrong choice. Thoughts?
          If there is an agreement, I propose to disable this in semantic analysis phase itself instead of letting it fail at runtime later.

          Show
          Ashutosh Chauhan added a comment - Thinking more about this. I dont think we want to support join on union type, since in general real data in a union could be of any type which makes it impossible to do comparison of union types correctly. It can be argued that in some special cases that could be achieved via type widening, but I would argue selecting union type for storing such data which later need to be joined is a wrong choice. Thoughts? If there is an agreement, I propose to disable this in semantic analysis phase itself instead of letting it fail at runtime later.
          Carl Steinbach made changes -
          Field Original Value New Value
          Link This issue relates to HIVE-2390 [ HIVE-2390 ]
          Carl Steinbach made changes -
          Labels uniontype
          Carl Steinbach made changes -
          Component/s Query Processor [ 12312586 ]
          Hide
          Chaoyu Tang added a comment -

          I believe the observed error was due to the lack of union type support in LazyBinarySerDe which is used to deserialize the reduce values in Join, rather than the join on the key of union type. So any join query with select values having union type (e.g. SELECT * FROM DEST1 JOIN DEST2 on (DEST1.value = DEST2.value) should failed with same NPE.

          Show
          Chaoyu Tang added a comment - I believe the observed error was due to the lack of union type support in LazyBinarySerDe which is used to deserialize the reduce values in Join, rather than the join on the key of union type. So any join query with select values having union type (e.g. SELECT * FROM DEST1 JOIN DEST2 on (DEST1.value = DEST2.value) should failed with same NPE.

            People

            • Assignee:
              Unassigned
              Reporter:
              Ashutosh Chauhan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development