Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1981

Improve getSplits performance by using listLocatedStatus

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.23.0
    • 0.23.10, 2.1.1-beta
    • job submission
    • None
    • Reviewed

    Description

      This jira will make FileInputFormat and CombinedFileInputForm to use the new API, thus reducing the number of RPCs to HDFS NameNode.

      Attachments

        1. mapredListFiles.patch
          16 kB
          Hairong Kuang
        2. mapredListFiles1.patch
          15 kB
          Hairong Kuang
        3. mapredListFiles2.patch
          15 kB
          Hairong Kuang
        4. mapredListFiles3.patch
          14 kB
          Hairong Kuang
        5. mapredListFiles4.patch
          15 kB
          Hairong Kuang
        6. mapredListFiles5.patch
          14 kB
          Hairong Kuang
        7. MAPREDUCE-1981.patch
          26 kB
          Jason Darrell Lowe
        8. MAPREDUCE-1981.branch-0.23.patch
          26 kB
          Jason Darrell Lowe

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5070 Implement listLocatedStatus() in ProxyFileSystem for 0.23 shim

          • Major - Major loss of function.
          • Resolved
          is blocked by

          New Feature - A new feature of the product, which has yet to be developed. HDFS-202 Add a bulk FIleSystem.getFileBlockLocations

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6219 Reduce memory required for FileInputFormat located status optimization

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-5603 Ability to disable FileInputFormat listLocatedStatus optimization to save client memory

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              hairong Hairong Kuang
              hairong Hairong Kuang
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: