[MAPREDUCE-7278] Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0, 3.4.0
• Fix Version/s: 3.3.0, 3.2.2, 3.4.0
• Component/s: task
• Assignee: Tarun Parimi
• Reporter: Tarun Parimi
• Created: 30/Apr/20 17:54
• Updated: 10/Jun/21 11:10
• Resolved: 28/May/20 09:47

Description

When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER state for some time, we observe two task attempts are launched simultaneously even when speculative execution is disabled.

This results in the below message shown in the killed attempts, indicating speculation has occurred. This is an issue for jobs which require speculative execution to be strictly disabled.

 

 

 

Issue Links

• is caused by: MAPREDUCE-6485 MR job hanged forever because all resources are taken up by reducers and the last map attempt never get resource to run