[YARN-1771] many getFileStatus calls made from node manager for localizing a public distributed cache resource

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: nodemanager
• Target Version/s: 2.4.0
• Assignee: Sangjin Lee
• Reporter: Sangjin Lee
• Created: 28/Feb/14 22:35
• Updated: 03/Sep/14 23:44
• Resolved: 14/Mar/14 00:31

Description

We're observing that the getFileStatus calls are putting a fair amount of load on the name node as part of checking the public-ness for localizing a resource that belong in the public cache.

We see 7 getFileStatus calls made for each of these resource. We should look into reducing the number of calls to the name node. One example:

2014-02-27 18:07:27,351 INFO audit: ... cmd=getfileinfo	src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...
2014-02-27 18:07:27,352 INFO audit: ... cmd=getfileinfo	src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...
2014-02-27 18:07:27,352 INFO audit: ... cmd=getfileinfo	src=/tmp/temp-887708724/tmp883330348 ...
2014-02-27 18:07:27,353 INFO audit: ... cmd=getfileinfo	src=/tmp/temp-887708724 ...
2014-02-27 18:07:27,353 INFO audit: ... cmd=getfileinfo	src=/tmp ...
2014-02-27 18:07:27,354 INFO audit: ... cmd=getfileinfo	src=/	 ...
2014-02-27 18:07:27,354 INFO audit: ... cmd=getfileinfo	src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...
2014-02-27 18:07:27,355 INFO audit: ... cmd=open	src=/tmp/temp-887708724/tmp883330348/foo-0.0.44.jar ...

Issue Links

• is related to: HDFS-6045 A single RPC API: FileStatus[] getFileStatus(Path f) to get status of all path components.