[YARN-1799] Enhance LocalDirAllocator in NM to consider DiskMaxUtilization cutoff

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.3.0
• Component/s: nodemanager
• Assignee: Sunil G
• Reporter: Sunil G
• Created: 07/Mar/14 04:59
• Updated: 15/May/14 11:19

Description

LocalDirAllocator provides paths for all tasks for its local write.
This considers the good list of directories which are selected by the HealthCheck mechamnism in LocalDirsHandlerService

getLocalPathForWrite() considers whether input demand "size" can meet the "capacity" in "lastAccessed" directory.
If more tasks asks for path from LocalDirAllocator, then it is possible that the allocation is done based on the current disk availability at that given time.
But this path would have earlier given to some other tasks to write and they may be sequentially doing writing.

It is better to check for an upper cutoff for disk availability