[MAPREDUCE-5907] Improve getSplits() performance for fs implementations that can utilize performance gains from recursive listing

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.0
• Component/s: client
• Assignee: Sumit Kumar
• Reporter: Sumit Kumar
• Created: 28/May/14 19:54
• Updated: 05/Jan/22 17:16

Description

FileInputFormat (both mapreduce and mapred implementations) use recursive listing while calculating splits. They however do this by doing listing level by level. That means to discover files in /foo/bar means they do listing at /foo/bar first to get the immediate children, then make the same call on all immediate children for /foo/bar to discover their immediate children and so on. This doesn't scale well for object store based fs implementations like s3 and swift because every listStatus call ends up being a webservice call to backend. In cases where large number of files are considered for input, this makes getSplits() call slow.

This patch adds a new set of recursive list apis that gives opportunity to the fs implementations to optimize. The behavior remains the same for other implementations (that is a default implementation is provided for other fs so they don't have to implement anything new). However for objectstore based fs implementations it provides a simple change to include recursive flag as true (as shown in the patch) to improve listing performance.

Issue Links

• depends upon: HADOOP-10634 Add recursive list apis to FileSystem to give implementations an opportunity for optimization
• is depended upon by: HADOOP-14302 Test MR split optimisation with recursive listing
• is depended upon by: HADOOP-16829 Über-jira: S3A Hadoop 3.3.1 features
• is related to: MAPREDUCE-7092 MR examples to work better against cloud stores