Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1981

Improve getSplits performance by using listLocatedStatus

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.10, 2.1.1-beta
    • Component/s: job submission
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      This jira will make FileInputFormat and CombinedFileInputForm to use the new API, thus reducing the number of RPCs to HDFS NameNode.

        Attachments

        1. mapredListFiles.patch
          16 kB
          Hairong Kuang
        2. mapredListFiles1.patch
          15 kB
          Hairong Kuang
        3. mapredListFiles2.patch
          15 kB
          Hairong Kuang
        4. mapredListFiles3.patch
          14 kB
          Hairong Kuang
        5. mapredListFiles4.patch
          15 kB
          Hairong Kuang
        6. mapredListFiles5.patch
          14 kB
          Hairong Kuang
        7. MAPREDUCE-1981.patch
          26 kB
          Jason Darrell Lowe
        8. MAPREDUCE-1981.branch-0.23.patch
          26 kB
          Jason Darrell Lowe

          Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5070 Implement listLocatedStatus() in ProxyFileSystem for 0.23 shim

          • Major - Major loss of function.
          • Resolved
          is blocked by

          New Feature - A new feature of the product, which has yet to be developed. HDFS-202 Add a bulk FIleSystem.getFileBlockLocations

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6219 Reduce memory required for FileInputFormat located status optimization

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-5603 Ability to disable FileInputFormat listLocatedStatus optimization to save client memory

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

            Activity

              People

              • Assignee:
                hairong Hairong Kuang
                Reporter:
                hairong Hairong Kuang
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: