Aaah....got the problem
The root cause for problem is:
Before scheduling reduces a check is made in JobInProgress: boolean org.apache.hadoop.mapred.JobInProgress.scheduleReduces() which makes the following check:
finishedMapTasks >= completedMapsForReduceSlowstart
This check is valid if the value of mapred.max.map.failures.percent is set to 0(zero/default value) but, when this value is set, the a/m check is invalid.
Say for example a Job spawns 100 Map task and the property value for mapred.max.map.failures.percent is set to 5 percent. In this scenario even if 95 Maps are successful, reducers should be scheduled. Now if we look back the a/m check, then the condition will not satisfy ever(if 5 map task fail) because
95 >= 100 will be always false.
As per my understanding the issue has nothing to with mapreduce.job.committer.setup.cleanup.needed.
We can't call the void org.apache.hadoop.mapred.JobInProgress.jobComplete() from the void org.apache.hadoop.mapred.JobInProgress.failedTask(TaskInProgress tip, TaskAttemptID taskid, TaskStatus status, TaskTracker taskTracker, boolean wasRunning, boolean wasComplete, boolean wasAttemptRunning) as the method will called upon failure of a Task but, we need to wait till completedMapsForReduceSlowstart is reached,so that reduces are spawned.
In my scenario there was a job(wordcount) with 111 Mappers.
The value of mapred.reduce.slowstart.completed.maps was set to 1 (100%). The value for mapreduce.map.failures.maxpercent was set to 5(5%).The Mapper implementation was tweaked in such a way that 4 mappers failed.
After some time I noticed that 107 mappers got completed but, reduces are not running and it got stuck for indefinite time.