[YARN-3491] PublicLocalizer#addResource is too slow.

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: nodemanager
• Hadoop Flags: Reviewed
• Assignee: Zhihai Xu
• Reporter: Zhihai Xu
• Created: 15/Apr/15 18:26
• Updated: 26/Feb/20 05:29
• Resolved: 06/May/15 21:21

Description

Based on the profiling, The bottleneck in PublicLocalizer#addResource is getInitializedLocalDirs. getInitializedLocalDirs call checkLocalDir.
checkLocalDir is very slow which takes about 10+ ms.
The total delay will be approximately number of local dirs * 10+ ms.
This delay will be added for each public resource localization.
Because PublicLocalizer#addResource is slow, the thread pool can't be fully utilized. Instead of doing public resource localization in parallel(multithreading), public resource localization is serialized most of the time.

And also PublicLocalizer#addResource is running in Dispatcher thread,
So the Dispatcher thread will be blocked by PublicLocalizer#addResource for long time.

Issue Links

• is related to: YARN-3549 use JNI-based FileStatus implementation from io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation from RawLocalFileSystem in checkLocalDir.
• relates to: YARN-3496 Add a configuration to disable/enable storing localization state in NMLeveldbStateStore