[MAPREDUCE-3808] NPE in FileOutputCommitter when running a 0 reduce job

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.23.0, 2.0.0-alpha
• Fix Version/s: 0.23.1
• Component/s: mrv2
• Target Version/s: 0.23.1
• Hadoop Flags: Reviewed
• Release Note: Fixed an NPE in FileOutputCommitter for jobs with maps but no reduces.
• Assignee: Robert Joseph Evans
• Reporter: Siddharth Seth
• Created: 04/Feb/12 19:51
• Updated: 10/Mar/15 04:32
• Resolved: 06/Feb/12 22:17

Description

This was while running LoadGen.

Error: java.lang.NullPointerException at org.apache.hadoop.fs.Path.<init>(Path.java:67) 
at org.apache.hadoop.fs.Path.<init>(Path.java:56) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185) 
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209) 
at org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100) 
at org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94) 
at org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176) 
at org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248) 
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955) 
at org.apache.hadoop.mapred.Task.done(Task.java:912) 
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331) 
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:396) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157) 
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)

Issue Links

• blocks: MAPREDUCE-3524 Scan benchmark is more than 1.5x slower in 0.23
• duplicates: MAPREDUCE-3821 NPE while running Shuffle benchmark
• is broken by: MAPREDUCE-3711 AppMaster recovery for Medium to large jobs take long time