[MAPREDUCE-1501] FileInputFormat to support multi-level/recursive directory listing

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.21.1
• Labels: inputformat
• Hadoop Flags: Reviewed
• Assignee: Zheng Shao
• Reporter: Zheng Shao
• Created: 17/Feb/10 22:55
• Updated: 13/Sep/10 06:11
• Resolved: 09/Mar/10 01:57

Description

As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.

4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/[e-mail]%3E

2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/[e-mail]%3E

6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/[e-mail]%3E

One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.

We can easily provide a JobConf option (which defaults to false) to FileInputFormat.listStatus(...) to recursively go into directory structure.

Issue Links

• blocks: HIVE-1083 allow sub-directories for an external table/partition
• relates to: MAPREDUCE-1577 FileInputFormat in the new mapreduce package to support multi-level/recursive directory listing