Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14557

Nullpointer When both SkewJoin and Mapjoin Enabled

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0, 2.1.0
    • Fix Version/s: 4.0.0
    • Component/s: Physical Optimizer
    • Labels:
      None

      Description

      The following sql failed with return code 2 on mr.

      create table a(id int,id1 int);
      create table b(id int,id1 int);
      create table c(id int,id1 int);
      set hive.optimize.skewjoin=true;
      select a.id,b.id,c.id1 from a,b,c where a.id=b.id and a.id1=c.id1;
      

      Error log as follows:

      2016-08-17 21:13:42,081 INFO [main] org.apache.hadoop.hive.ql.exec.mr.ExecMapper: 
      <MAP>Id =0
        <Children>
          <TS>Id =21
            <Children>
              <MAPJOIN>Id =28
                <Children>
                  <FS>Id =16
                    <Children>
                    <\Children>
                    <Parent>Id = 28 null<\Parent>
                  <\FS>
                <\Children>
                <Parent>Id = 21 nullId = 33 
              <HASHTABLEDUMMY>Id =33
                <Children>null
                <\Children>
                <Parent><\Parent>
              <\HASHTABLEDUMMY><\Parent>
              <\MAPJOIN>
            <\Children>
            <Parent>Id = 0 null<\Parent>
          <\TS>
        <\Children>
        <Parent><\Parent>
      <\MAP>
      2016-08-17 21:13:42,084 INFO [main] org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[21]
      2016-08-17 21:13:42,084 INFO [main] org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Initializing dummy operator
      2016-08-17 21:13:42,086 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:0, RECORDS_IN:0, 
      2016-08-17 21:13:42,087 ERROR [main] org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Hit error while closing operators - failing tree
      2016-08-17 21:13:42,088 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: Hive Runtime Error while closing operators
      	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
      	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
      	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
      Caused by: java.lang.NullPointerException
      	at org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:474)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:682)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
      	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
      	... 8 more
      
      

        Attachments

        1. HIVE-14557.patch
          1 kB
          Nemon Lou
        2. HIVE-14557.2.patch
          21 kB
          Rui Li
        3. HIVE-14557.3.patch
          24 kB
          Rui Li
        4. HIVE-14557.3.patch
          24 kB
          Rui Li

          Issue Links

            Activity

              People

              • Assignee:
                lirui Rui Li
                Reporter:
                nemon Nemon Lou
              • Votes:
                1 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: