Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5756

CombineFileInputFormat.getSplits() including directories in its results

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Trying to track down HIVE-6401, where we see some "is not a file" errors because getSplits() is giving us directories. I believe the culprit is FileInputFormat.listStatus():

                      if (recursive && stat.isDirectory()) {
                        addInputPathRecursively(result, fs, stat.getPath(),
                            inputFilter);
                      } else {
                        result.add(stat);
                      }
      

      Which seems to be allowing directories to be added to the results if recursive is false. Is this meant to return directories? If not, I think it should look like this:

                      if (stat.isDirectory()) {
                       if (recursive) {
                        addInputPathRecursively(result, fs, stat.getPath(),
                            inputFilter);
                       }
                      } else {
                        result.add(stat);
                      }
      

        Attachments

        1. MAPREDUCE-5756.2.patch
          4 kB
          Jason Dere
        2. MAPREDUCE-5756.1.patch
          1.0 kB
          Jason Dere

          Issue Links

            Activity

              People

              • Assignee:
                jdere Jason Dere
                Reporter:
                jdere Jason Dere
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: