[MAPREDUCE-6721] mapreduce.reduce.shuffle.memory.limit.percent=0.0 should be legal to enforce shuffle to disk

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.4
• Fix Version/s: 2.9.0, 3.0.0-alpha1
• Component/s: mrv2, task
• Target Version/s: 2.9.0
• Hadoop Flags: Reviewed
• Assignee: Gera Shegalov
• Reporter: Gera Shegalov
• Created: 22/Jun/16 07:18
• Updated: 30/Aug/16 01:13
• Resolved: 23/Jun/16 00:24

Description

We are potentially hitting an in-memory-shuffle-related reservation starvation resembling MAPREDUCE-6445. To work it around, we wanted to disable in memory shuffle via mapreduce.reduce.shuffle.memory.limit.percent=0.0 that turned out to be disallowed by the current logic. So we had to resort to another small float value such as 0.0001. However, zero is more logical imo.

Issue Links

• relates to: MAPREDUCE-6445 Shuffle hang
• links to: GitHub Pull Request #102