[PIG-2856] AvroStorage doesn't load files in the directories when a glob pattern matches both files and directories.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.11
• Fix Version/s: 0.11
• Component/s: piggybank
• Patch Info: Patch Available
• Assignee: Cheolsoo Park
• Reporter: Cheolsoo Park
• Created: 02/Aug/12 09:25
• Updated: 22/Feb/13 04:54
• Resolved: 06/Aug/12 05:22

Description

This is a regression from PIG-2492.

When a glob pattern such as '*' matches not only files but also directories, AvroStorage does not load files in the directories. This is a bug in getAllSubDirs() that can be fixed as follows:

static boolean getAllSubDirs(Path path, Job job, Set<Path> paths)
...
FileStatus[] matchedFiles = fs.globStatus(path, PATH_FILTER);
...
for (FileStatus file : matchedFiles) {
    if (file.isDir()) {
-        for (FileStatus sub : fs.listStatus(path)) {
+        for (FileStatus sub : fs.listStatus(file.getPath())) {
            getAllSubDirs(sub.getPath(), job, paths);
        }
    }
}