Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5756

CombineFileInputFormat.getSplits() including directories in its results

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.6.0
    • None
    • None
    • Reviewed

    Description

      Trying to track down HIVE-6401, where we see some "is not a file" errors because getSplits() is giving us directories. I believe the culprit is FileInputFormat.listStatus():

                      if (recursive && stat.isDirectory()) {
                        addInputPathRecursively(result, fs, stat.getPath(),
                            inputFilter);
                      } else {
                        result.add(stat);
                      }
      

      Which seems to be allowing directories to be added to the results if recursive is false. Is this meant to return directories? If not, I think it should look like this:

                      if (stat.isDirectory()) {
                       if (recursive) {
                        addInputPathRecursively(result, fs, stat.getPath(),
                            inputFilter);
                       }
                      } else {
                        result.add(stat);
                      }
      

      Attachments

        1. MAPREDUCE-5756.1.patch
          1.0 kB
          Jason Dere
        2. MAPREDUCE-5756.2.patch
          4 kB
          Jason Dere

        Issue Links

          Activity

            People

              jdere Jason Dere
              jdere Jason Dere
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: