[TEZ-2271] OutputFormat is not recognized correctly

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Target Version/s: 0.5.4
• Assignee: Jeff Zhang
• Reporter: Jeff Zhang
• Created: 03/Apr/15 06:04
• Updated: 29/Apr/15 05:53
• Resolved: 29/Apr/15 05:53

Description

TEZ-2162 didn't resolve it completely. Paste the comment from TEZ-2162

But there's one exceptional case that if user specify OutputFormat through conf, then even MROutput is on mapper side, user still need to use property MRJobConfig.NEW_API_REDUCER_CONFIG rather than NEW_API_MAPPER_CONFIG. Or I think we can take useNewAPI as true as any of NEW_API_MAPPER_CONFIG, NEW_API_REDUCER_CONFIG is set true (patch didn't include this now)

This issue happens when translating from Mapper-only MR job to tez.

12:50:07,425 - Thread( main) - (Job.java:1386) - Job job_1427876879456_0010 failed with state FAILED due to: Vertex failed, vertexName=initialmap, vertexId=vertex_1427876879456_0010_1_00, diagnostics=[Task failed, taskId=task_1427876879456_0010_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.ClassCastException: org.apache.hadoop.mapred.FileOutputCommitter cannot be cast to org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
	at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getWorkOutputPath(FileOutputFormat.java:222)
	at org.apache.sqoop.mapreduce.TextImportMapper.setup(TextImportMapper.java:48)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
	at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
	at org.apache.tez.mapreduce.processor.map.MapProcessor.runNewMapper(MapProcessor.java:237)
	at org.apache.tez.mapreduce.processor.map.MapProcessor.run(MapProcessor.java:124)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:330)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:180)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Issue Links

• relates to: TEZ-2305 MR compatibility sleep job fails with IOException: Undefined job output-path