[MAPREDUCE-5756] CombineFileInputFormat.getSplits() including directories in its results

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0
• Target Version/s: 2.6.0
• Hadoop Flags: Reviewed
• Assignee: Jason Dere
• Reporter: Jason Dere
• Created: 12/Feb/14 19:55
• Updated: 16/Dec/14 03:02
• Resolved: 21/Jul/14 21:30

Description

Trying to track down HIVE-6401, where we see some "is not a file" errors because getSplits() is giving us directories. I believe the culprit is FileInputFormat.listStatus():

                if (recursive && stat.isDirectory()) {
                  addInputPathRecursively(result, fs, stat.getPath(),
                      inputFilter);
                } else {
                  result.add(stat);
                }

Which seems to be allowing directories to be added to the results if recursive is false. Is this meant to return directories? If not, I think it should look like this:

                if (stat.isDirectory()) {
                 if (recursive) {
                  addInputPathRecursively(result, fs, stat.getPath(),
                      inputFilter);
                 }
                } else {
                  result.add(stat);
                }

Issue Links

• blocks: HIVE-6401 root_dir_external_table.q fails with -Phadoop-2