[MAPREDUCE-5703] Job client gets failure though RM side job execution result is FINISHED and SUCCEEDED

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Component/s: client
• Assignee: Unassigned
• Reporter: Ashutosh Jindal
• Created: 31/Dec/13 10:47
• Updated: 09/May/15 00:41
• Resolved: 09/May/15 00:41

Description

1) Run MR job
2) After reduce completed and while JHS file writing, restart DN.

RM side job is shown as successful.
JHS doesnt have info about the job.
Job client gets NPE and exit code as 255.

java.io.IOException: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:269)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173)
at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:929)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2080)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2076)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2074)

at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:330)
at org.apache.hadoop.mapred.ClientServiceDelegate.getTaskCompletionEvents(ClientServiceDelegate.java:382)
at org.apache.hadoop.mapred.YARNRunner.getTaskCompletionEvents(YARNRunner.java:529)
at org.apache.hadoop.mapreduce.Job$5.run(Job.java:668)
at org.apache.hadoop.mapreduce.Job$5.run(Job.java:665)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:665)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1349)
at org.apache.hadoop.mapred.JobClient$NetworkedJob.monitorAndPrintJob(JobClient.java:407)
at org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:855)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:835)

Issue Links

• is related to: MAPREDUCE-5547 Job history should not be flushed to JHS until AM gets unregistered