Pig
  1. Pig
  2. PIG-2856

AvroStorage doesn't load files in the directories when a glob pattern matches both files and directories.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.11
    • Fix Version/s: 0.11
    • Component/s: piggybank
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      This is a regression from PIG-2492.

      When a glob pattern such as '*' matches not only files but also directories, AvroStorage does not load files in the directories. This is a bug in getAllSubDirs() that can be fixed as follows:

      static boolean getAllSubDirs(Path path, Job job, Set<Path> paths)
      ...
      FileStatus[] matchedFiles = fs.globStatus(path, PATH_FILTER);
      ...
      for (FileStatus file : matchedFiles) {
          if (file.isDir()) {
      -        for (FileStatus sub : fs.listStatus(path)) {
      +        for (FileStatus sub : fs.listStatus(file.getPath())) {
                  getAllSubDirs(sub.getPath(), job, paths);
              }
          }
      }
      
      1. PIG-2856-2.patch
        3 kB
        Cheolsoo Park
      2. PIG-2856.patch
        2 kB
        Cheolsoo Park

        Activity

        Cheolsoo Park created issue -
        Cheolsoo Park made changes -
        Field Original Value New Value
        Attachment PIG-2856.patch [ 12538887 ]
        Cheolsoo Park made changes -
        Affects Version/s 0.11 [ 12318878 ]
        Affects Version/s 0.10.0 [ 12316246 ]
        Patch Info Patch Available [ 10042 ]
        Cheolsoo Park made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Cheolsoo Park made changes -
        Attachment PIG-2856-2.patch [ 12539174 ]
        Santhosh Srinivasan made changes -
        Fix Version/s 0.11 [ 12318878 ]
        Santhosh Srinivasan made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Bill Graham made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Cheolsoo Park
            Reporter:
            Cheolsoo Park
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development