Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12810

FileSystem#listLocatedStatus causes unnecessary RPC calls

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
    • Component/s: fs, fs/s3
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      FileSystem#listLocatedStatus lists the files in a directory and then calls getFileBlockLocations(stat.getPath(), ...) for each instead of getFileBlockLocations(stat, ...). That function with the path arg just calls getFileStatus to get another file status from the path and calls the file status version, so this ends up calling getFileStatus unnecessarily.

      This is particularly bad for S3, where getFileStatus is expensive. Avoiding the extra call improved input split calculation time for a data set in S3 by ~20x: from 10 minutes to 25 seconds.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rdblue Ryan Blue
                Reporter:
                rdblue Ryan Blue
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: