Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9863

Randomize List of Resources to Localize

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: nodemanager
    • Labels:
      None

      Description

      https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/LocalResourceBuilder.java

      Add a new parameter to LocalResourceBuilder that allows the list of resources to be shuffled randomly. This will allow the Localizer to spread the load of requests so that not all of the NodeManagers are requesting to localize the same files, in the same order, from the same DataNodes,

        Attachments

        1. YARN-9863.2.patch
          3 kB
          David Mollitor
        2. YARN-9863.1.patch
          3 kB
          David Mollitor

          Activity

            People

            • Assignee:
              belugabehr David Mollitor
              Reporter:
              belugabehr David Mollitor
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: