Hive
  1. Hive
  2. HIVE-4650

Getting Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask on auto convert to MapJoin after upgrade to Hive-0.11.0.x from hive-0.10.0.x

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: Query Processor
    • Labels:
      None
    • Environment:

      HortonWorks 1.3 distro on x86_64 Centos 6

      Description

      working from a simple table in Hive

      hive> desc cmnt
      > ;
      OK
      x1 int None
      x2 int None
      x3 int None
      x4 int None
      y double None

      hive> select * from cmnt;
      OK
      7 26 6 60 78.5
      1 29 15 52 74.3
      11 56 8 20 104.3
      11 31 8 47 87.6
      7 52 6 33 95.9
      11 55 9 22 109.2
      3 71 17 6 102.7
      1 31 22 44 72.5
      2 54 18 22 93.1
      21 47 4 26 115.9
      1 40 23 34 83.8
      11 66 9 12 113.3
      10 68 8 12 109.4

      A query that joins and transforms against this table :

      select * from (select VAL001 x1,VAL002 x2,VAL003 x3,VAL004 x4,VAL005 y from ( select /*+ mapjoin(v2) */ (VAL001- mu1) * 1/(sd1) VAL001,(VAL002- mu2) * 1/(sd2) VAL002,(VAL003- mu3) * 1/(sd3) VAL003,(VAL004- mu4) * 1/(sd4) VAL004,(VAL005- mu5) * 1/(sd5) VAL005 from ( select * from ( select x1 VAL001,x2 VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3 ) v3 join (select count c, avg(VAL001) mu1,avg(VAL002) mu2,avg(VAL003) mu3,avg(VAL004) mu4,avg(VAL005) mu5, stddev_pop(VAL001) sd1,stddev_pop(VAL002) sd2,stddev_pop(VAL003) sd3,stddev_pop(VAL004) sd4,stddev_pop(VAL005) sd5 from ( select * from ( select x1 VAL001,x2 VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3 ) v1) v2 ) obj1_7) obj1_6 ;

      Generates during Stage-3 :
      setting HADOOP_USER_NAME test
      Execution log at: /tmp/test/.log
      2013-06-03 12:40:55 Starting to launch local task to process map join; maximum memory = 1065484288
      2013-06-03 12:40:56 Processing rows: 1 Hashtable size: 1 Memory usage: 7175528 rate: 0.007
      2013-06-03 12:40:56 Dump the hashtable into file: file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
      2013-06-03 12:40:56 Upload 1 File to: file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable File size: 334
      2013-06-03 12:40:56 End of local task; Time Taken: 0.726 sec.
      Execution completed successfully
      Mapred Local Task Succeeded . Convert the Join into MapJoin
      Mapred Local Task Succeeded . Convert the Join into MapJoin
      Launching Job 2 out of 2
      Number of reduce tasks is set to 0 since there's no reduce operator
      Starting Job = job_201306022123_0045, Tracking URL = http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
      Kill Command = /usr/lib/hadoop/libexec/../bin/hadoop job -kill job_201306022123_0045
      Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 0
      2013-06-03 00:41:05,895 Stage-3 map = 0%, reduce = 0%
      2013-06-03 00:41:40,687 Stage-3 map = 100%, reduce = 100%
      Ended Job = job_201306022123_0045 with errors
      Error during job, obtaining debugging information...
      Job Tracking URL: http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
      Examining task ID: task_201306022123_0045_m_000002 (and more) from job job_201306022123_0045

      Task with the most failures(4):


      Task ID:
      task_201306022123_0045_m_000000

      URL:
      http://sun1vm3:50030/taskdetails.jsp?jobid=job_201306022123_0045&tipid=task_201306022123_0045_m_000000


      Diagnostic Messages for this Task:
      java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
      at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
      at org.apache.hadoop.mapred.Child.main(Child.java:249)
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
      at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198)
      at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
      at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1377)
      at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
      at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
      at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:611)
      at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
      ... 8 more
      Caused by: java.lang.NullPointerException
      at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:186)
      ... 14 more

      FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask

      If "hive.auto.convert.join = false" is set then the all the query stages work OK. The same scenario worked OK in Hive-0.10.0.x and Hive-0.9.x with MapJoin working.

        Issue Links

          Activity

          Hide
          Ashutosh Chauhan added a comment -

          Bruce,
          I will encourage you to ask this question on Hortonworks mailing list which is the best place to ask questions about Hortonworks distribution.
          The next apache release of Hive will definitely contain this fix.

          Show
          Ashutosh Chauhan added a comment - Bruce, I will encourage you to ask this question on Hortonworks mailing list which is the best place to ask questions about Hortonworks distribution. The next apache release of Hive will definitely contain this fix.
          Hide
          Bruce Nelson added a comment -

          Ashtoush

          Thanks - I was going to write and see if we have a resolution out of 4502. Will we see a patch in Hortonworks on 0.11 - looks like 0.12 will be the fix.

          Thanks

          Bruce Nelson | Principal Sales Consultant
          Big Data Area Lead West
          Hadoop noSQL Big Data Specialist
          Phone: +1 8188173046 | Mobile: +1 3102667111
          Oracle North America Technology Organization
          15760 Ventura Boulevard Siute 1400 | Encino, California 91436

          Oracle is committed to developing practices and products that help protect the environment

          Show
          Bruce Nelson added a comment - Ashtoush Thanks - I was going to write and see if we have a resolution out of 4502. Will we see a patch in Hortonworks on 0.11 - looks like 0.12 will be the fix. Thanks – Bruce Nelson | Principal Sales Consultant Big Data Area Lead West Hadoop noSQL Big Data Specialist Phone: +1 8188173046 | Mobile: +1 3102667111 Oracle North America Technology Organization 15760 Ventura Boulevard Siute 1400 | Encino, California 91436 Oracle is committed to developing practices and products that help protect the environment
          Hide
          Ashutosh Chauhan added a comment -

          HIVE-4502 has been checked in which is suppose to fix the issue. If you can still reproduce feel free to reopen.

          Show
          Ashutosh Chauhan added a comment - HIVE-4502 has been checked in which is suppose to fix the issue. If you can still reproduce feel free to reopen.
          Hide
          Vikram Dixit K added a comment -

          Typo. I mean it will be updated.


          Nothing better than when appreciated for hard work.
          -Mark

          Show
          Vikram Dixit K added a comment - Typo. I mean it will be updated. – Nothing better than when appreciated for hard work. -Mark
          Hide
          Vikram Dixit K added a comment -

          Bruce,

          I have already done that. Once I have the test results, it will be
          committed.

          Thanks
          Vikram.


          Nothing better than when appreciated for hard work.
          -Mark

          Show
          Vikram Dixit K added a comment - Bruce, I have already done that. Once I have the test results, it will be committed. Thanks Vikram. – Nothing better than when appreciated for hard work. -Mark
          Hide
          Bruce Nelson added a comment -

          Vikram: I was looking at D10695 and I can how it would work - since patch HIVE-4502-1 is already applied to HWORX's 1.3.0 code I will sharpen my pencil and see what I can do and publish it back to this JIRA.

          Show
          Bruce Nelson added a comment - Vikram: I was looking at D10695 and I can how it would work - since patch HIVE-4502 -1 is already applied to HWORX's 1.3.0 code I will sharpen my pencil and see what I can do and publish it back to this JIRA.
          Hide
          Vikram Dixit K added a comment -

          Bruce: I meant the patch HIVE-4502.D10695.patch is the one that fixes the issue. HIVE-4502-1.patch is not going to be the patch for it as it does not fix this case. Those differences that you mention is because of the inherent version differences and will need to be resolved. I have attached the results from a test I ran to verify a case similar to yours.

          Show
          Vikram Dixit K added a comment - Bruce: I meant the patch HIVE-4502 .D10695.patch is the one that fixes the issue. HIVE-4502 -1.patch is not going to be the patch for it as it does not fix this case. Those differences that you mention is because of the inherent version differences and will need to be resolved. I have attached the results from a test I ran to verify a case similar to yours.
          Hide
          Bruce Nelson added a comment -

          Correction - typo on my part its hive-0.11.0.1.3.0.0-107 not *-170

          Show
          Bruce Nelson added a comment - Correction - typo on my part its hive-0.11.0.1.3.0.0-107 not *-170
          Hide
          Bruce Nelson added a comment -

          Vikram : I took a look at JIRA HIVE-4502 and Hortonworks hive-0.11.0.1.3.0.0-170 (HWorx 1.3.0) has HIVE-4502-1.patch already applied in the src. I did verify compiling the source and running the May 20th hive-0.11.0 package and the same failure exists. The HIVE-4502.D10695.patch is not possible because of differences in code. I will report this on JIRA HIVE-4502 as well. So it appears that 4502 is not a fix for this particular issue.

          Show
          Bruce Nelson added a comment - Vikram : I took a look at JIRA HIVE-4502 and Hortonworks hive-0.11.0.1.3.0.0-170 (HWorx 1.3.0) has HIVE-4502 -1.patch already applied in the src. I did verify compiling the source and running the May 20th hive-0.11.0 package and the same failure exists. The HIVE-4502 .D10695.patch is not possible because of differences in code. I will report this on JIRA HIVE-4502 as well. So it appears that 4502 is not a fix for this particular issue.
          Hide
          Vikram Dixit K added a comment -

          This is related to the issue HIVE-4502. I have verified that the plans generated for this query with the existing patch on HIVE-4502 look fine. I will close this based on the resolution of that issue.

          Show
          Vikram Dixit K added a comment - This is related to the issue HIVE-4502 . I have verified that the plans generated for this query with the existing patch on HIVE-4502 look fine. I will close this based on the resolution of that issue.
          Hide
          Bruce Nelson added a comment -

          If "hive.auto.convert.join = false" is set then the all the query stages work OK. The same scenario worked OK in Hive-0.10.0.x and Hive-0.9.x with MapJoin working.

          Show
          Bruce Nelson added a comment - If "hive.auto.convert.join = false" is set then the all the query stages work OK. The same scenario worked OK in Hive-0.10.0.x and Hive-0.9.x with MapJoin working.

            People

            • Assignee:
              Unassigned
              Reporter:
              Bruce Nelson
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development