[HIVE-16235] Hive queries fail with return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Assignee: Wei Zheng
• Reporter: Kavan Suresh
• Created: 16/Mar/17 22:22
• Updated: 17/Jan/19 11:31
• Resolved: 03/Apr/17 23:31

Description

Simplified version for reproducing issue as provided by wzheng

set hive.mapred.mode=nonstrict;
set hive.explain.user=false;
set hive.execution.engine=mr;
set hive.auto.convert.join=true;

DROP TABLE IF EXISTS t1;

CREATE TABLE t1 (c1 int, c2 int) clustered by (c1) into 10 buckets stored as orc;

INSERT INTO t1 VALUES (1, 2), (3, 4);

SELECT * FROM t1;

EXPLAIN
SELECT INPUT__FILE__NAME, t1.c1, t1.c2
FROM t1
INNER JOIN
(SELECT * FROM t1 WHERE t1.c1 = 1) sub
ON sub.c1 = t1.c1;

SELECT INPUT__FILE__NAME, t1.c1, t1.c2
FROM t1
INNER JOIN
(SELECT * FROM t1 WHERE t1.c1 = 1) sub
ON sub.c1 = t1.c1;

From hiveserver2.log:

INFO  [HiveServer2-Background-Pool: Thread-694]: lockmgr.DbTxnManager (DbTxnManager.java:acquireLocks(207)) - Setting lock request transaction to txnid:58 for queryId=hive_20170314035259_6a38c0fa-3dcd-4ebf-abc2-6ad61f33e040
....
 [HiveServer2-Background-Pool: Thread-694]: ql.Driver (Driver.java:execute(1411)) - Starting command(queryId=hive_20170314035259_6a38c0fa-3dcd-4ebf-abc2-6ad61f33e040): <query>
....
 ERROR [HiveServer2-Background-Pool: Thread-694]: exec.Task (SessionState.java:printError(993)) -
Task failed!
Task ID:
  Stage-15

Logs:
....
 ERROR [HiveServer2-Background-Pool: Thread-694]: ql.Driver (SessionState.java:printError(993)) - FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

There are no failed jobs on the RM UI, the MapredLocalTask log shows the below error:

ERROR mr.MapredLocalTask (MapredLocalTask.java:executeInProcess(356)) - Hive Runtime Error: Map local work failed
java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:56)
        at org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:77)
        at org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:147)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:363)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:482)
        at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:439)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
        at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:461)
        at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:365)
        at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:345)
        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:744)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

Issue Links

• is related to: HIVE-19286 NPE in MERGE operator on MR mode
• is related to: HIVE-19295 Some multiple inserts do not work on MR engine