[HADOOP-10048] LocalDirAllocator should avoid holding locks while accessing the filesystem

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Jason Darrell Lowe
• Reporter: Jason Darrell Lowe
• Created: 15/Oct/13 16:08
• Updated: 25/Oct/19 20:25
• Resolved: 07/Jun/16 16:27

Description

As noted in MAPREDUCE-5584 and HADOOP-7016, LocalDirAllocator can be a bottleneck for multithreaded setups like the ShuffleHandler. We should consider moving to a lockless design or minimizing the critical sections to a very small amount of time that does not involve I/O operations.

Issue Links

• relates to: MAPREDUCE-5584 ShuffleHandler becomes unresponsive during gridmix runs and can leak file descriptors
• relates to: HADOOP-7016 Optimize locking codepath in LocalDirAllocator.getLocalPathToRead() and reduce creating many Path objects