[MAPREDUCE-6870] Add configuration for MR job to finish when all reducers are complete (even with unfinished mappers)

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.1
• Fix Version/s: 3.0.0-beta1
• Target Version/s: 2.7.5
• Hadoop Flags: Incompatible change, Reviewed

Release Note

Enables mapreduce.job.finish-when-all-reducers-done by default. With this enabled, a MapReduce job will complete as soon as all of its reducers are complete, even if some mappers are still running. This can occur if a mapper was relaunched after node failure but the relaunched task's output is not actually needed. Previously the job would wait for all mappers to complete.

• Assignee: Peter Bacsko
• Reporter: Zhe Zhang
• Created: 28/Mar/17 18:45
• Updated: 11/Sep/17 21:22
• Resolved: 10/Aug/17 22:29

Description

Even with MAPREDUCE-5817, there could still be cases where mappers get scheduled before all reducers are complete, but those mappers run for long time, even after all reducers are complete. This could hurt the performance of large MR jobs.

In some cases, mappers don't have any materialize-able outcome other than providing intermediate data to reducers. In that case, the job owner should have the config option to finish the job once all reducers are complete.

Issue Links

• is related to: MAPREDUCE-6937 Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
• relates to: MAPREDUCE-6939 Follow-up on MAPREDUCE-6870
• relates to: MAPREDUCE-5817 Mappers get rescheduled on node transition even after all reducers are completed