Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1981

Improve getSplits performance by using listLocatedStatus

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.10, 2.1.1-beta
    • Component/s: job submission
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      This jira will make FileInputFormat and CombinedFileInputForm to use the new API, thus reducing the number of RPCs to HDFS NameNode.

        Attachments

        1. MAPREDUCE-1981.patch
          26 kB
          Jason Darrell Lowe
        2. MAPREDUCE-1981.branch-0.23.patch
          26 kB
          Jason Darrell Lowe
        3. mapredListFiles5.patch
          14 kB
          Hairong Kuang
        4. mapredListFiles4.patch
          15 kB
          Hairong Kuang
        5. mapredListFiles3.patch
          14 kB
          Hairong Kuang
        6. mapredListFiles2.patch
          15 kB
          Hairong Kuang
        7. mapredListFiles1.patch
          15 kB
          Hairong Kuang
        8. mapredListFiles.patch
          16 kB
          Hairong Kuang

          Issue Links

            Activity

              People

              • Assignee:
                hairong Hairong Kuang
                Reporter:
                hairong Hairong Kuang
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: