[HADOOP-12810] FileSystem#listLocatedStatus causes unnecessary RPC calls

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.2
• Fix Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
• Component/s: fs, fs/s3
• Target Version/s: 2.6.5
• Hadoop Flags: Reviewed
• Assignee: Ryan Blue
• Reporter: Ryan Blue
• Created: 16/Feb/16 22:28
• Updated: 06/Jan/17 01:46
• Resolved: 17/Feb/16 04:47

Description

FileSystem#listLocatedStatus lists the files in a directory and then calls getFileBlockLocations(stat.getPath(), ...) for each instead of getFileBlockLocations(stat, ...). That function with the path arg just calls getFileStatus to get another file status from the path and calls the file status version, so this ends up calling getFileStatus unnecessarily.

This is particularly bad for S3, where getFileStatus is expensive. Avoiding the extra call improved input split calculation time for a data set in S3 by ~20x: from 10 minutes to 25 seconds.

Issue Links

• breaks: MAPREDUCE-6637 Testcase Failure : TestFileInputFormat.testSplitLocationInfo
• is depended upon by: HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
• relates to: SPARK-9926 Parallelize file listing for partitioned Hive table
• supercedes: SPARK-10340 Use S3 bulk listing for S3-backed Hive tables