[MAPREDUCE-5448] MapFileOutputFormat#getReaders bug with invisible files/folders

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: mrv2
• Hadoop Flags: Reviewed
• Assignee: Maysam Yabandeh
• Reporter: Maysam Yabandeh
• Created: 05/Aug/13 08:28
• Updated: 30/Aug/16 01:20
• Resolved: 22/Mar/15 04:26

Description

MapReduce jobs also produce some invisible files such as _SUCCESS, even when the output format is MapFileOutputFormat. MapFileOutputFormat#getReaders however reads the entire content of the job output, assming that they are MapFiles.

Path[] names = FileUtil.stat2Paths(fs.listStatus(dir));

It should use a filter to skip the files that start with "." or "_".