[MAPREDUCE-5000] TaskImpl.getCounters() can return the counters for the wrong task attempt when task is speculating

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.23.6
• Fix Version/s: 0.23.7, 2.1.0-beta
• Component/s: mr-am
• Target Version/s: 0.23.7, 2.1.0-beta
• Hadoop Flags: Reviewed
• Assignee: Jason Darrell Lowe
• Reporter: Jason Darrell Lowe
• Created: 11/Feb/13 19:31
• Updated: 27/Aug/13 22:21
• Resolved: 13/Feb/13 19:28

Description

When a task is speculating and one attempt completes then sometimes the counters for the wrong attempt are aggregated into the total counters for the job. The scenario looks like this:

1. Two task attempts are racing, _0 and _1
2. _1 finishes first, causing the task to issue a TA_KILL to attempt _0
3. _0 receives TA_KILL, sets progress to 1.0f and waits for container cleanup
4. if TaskImpl.getCounters() is called now, TaskImpl.selectBestAttempt() can return _0 since it is not quite yet in the KILLED state yet progress is maxed out and no other attempt has more progress.