[TEZ-3293] Fetch failures can cause a shuffle hang waiting for memory merge that never starts

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.7.1, 0.8.3
• Fix Version/s: 0.7.2, 0.9.0, 0.8.4
• Target Version/s: 0.7.2, 0.8.4
• Assignee: Jason Darrell Lowe
• Reporter: Jason Darrell Lowe
• Created: 08/Jun/16 20:19
• Updated: 09/Jul/16 19:16
• Resolved: 30/Jun/16 23:28

Description

Tez jobs can hang in shuffle waiting for a memory merge that never starts. When a MapOutput is reserved it increments usedMemory but when it is unreserved it decrements usedMemory and commitMemory. If enough shuffle failures occur of sufficient size then commitMemory may never reach the merge threshold even after all outstanding transfers have committed and thus hang the shuffle.

Issue Links

• relates to: MAPREDUCE-4842 Shuffle race can hang reducer