Hadoop YARN
  1. Hadoop YARN
  2. YARN-291

[Umbrella] Dynamic resource configuration

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: nodemanager, scheduler
    • Labels:

      Description

      The current Hadoop YARN resource management logic assumes per node resource is static during the lifetime of the NM process. Allowing run-time configuration on per node resource will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist with other workloads on the same hardware efficiently, whether or not the environment is virtualized. More background and design details can be found in attached proposal.

      1. YARN-291-YARNClientCommandline-04.patch
        8 kB
        Junping Du
      2. YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch
        6 kB
        Junping Du
      3. YARN-291-JMXInterfaceOnNM-02.patch
        4 kB
        Junping Du
      4. YARN-291-core-HeartBeatAndScheduler-01.patch
        31 kB
        Junping Du
      5. YARN-291-CoreAndAdmin.patch
        46 kB
        Junping Du
      6. YARN-291-all-v1.patch
        66 kB
        Junping Du
      7. YARN-291-AddClientRMProtocolToSetNodeResource-03.patch
        25 kB
        Junping Du
      8. Elastic Resources for YARN-v0.2.pdf
        512 kB
        Junping Du

        Issue Links

          Activity

          Junping Du created issue -
          Junping Du made changes -
          Field Original Value New Value
          Link This issue is part of HADOOP-9165 [ HADOOP-9165 ]
          Junping Du made changes -
          Link This issue relates to MAPREDUCE-4660 [ MAPREDUCE-4660 ]
          Junping Du made changes -
          Link This issue relates to MAPREDUCE-4660 [ MAPREDUCE-4660 ]
          Junping Du made changes -
          Link This issue relates to MAPREDUCE-4900 [ MAPREDUCE-4900 ]
          Junping Du made changes -
          Attachment YARN-291-all-v1.patch [ 12562837 ]
          Junping Du made changes -
          Junping Du made changes -
          Attachment YARN-291-JMXInterfaceOnNM-02.patch [ 12562839 ]
          Junping Du made changes -
          Junping Du made changes -
          Attachment YARN-291-YARNClientCommandline-04.patch [ 12562841 ]
          Junping Du made changes -
          Junping Du made changes -
          Attachment Elastic Resources for YARN-v0.2.pdf [ 12562888 ]
          Luke Lu made changes -
          Link This issue is part of HADOOP-9165 [ HADOOP-9165 ]
          Luke Lu made changes -
          Link This issue supercedes HADOOP-9165 [ HADOOP-9165 ]
          Luke Lu made changes -
          Summary Dynamic resource configuration on NM Dynamic resource configuration
          Junping Du made changes -
          Link This issue is related to YARN-160 [ YARN-160 ]
          Junping Du made changes -
          Link This issue relates to YARN-45 [ YARN-45 ]
          Junping Du made changes -
          Link This issue relates to MAPREDUCE-4584 [ MAPREDUCE-4584 ]
          Junping Du made changes -
          Description The current Hadoop YARN resource management logic assumes per node resource is static during the lifetime of the NM process. Allowing run-time configuration on per node resource will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist with other workloads on the same hardware efficiently, whether or not the environment is virtualized. About more background and design details, please refer: HADOOP-9165. The current Hadoop YARN resource management logic assumes per node resource is static during the lifetime of the NM process. Allowing run-time configuration on per node resource will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist with other workloads on the same hardware efficiently, whether or not the environment is virtualized. More background and design details can be found in attached proposal.
          Luke Lu made changes -
          Link This issue blocks YARN-914 [ YARN-914 ]
          Junping Du made changes -
          Attachment YARN-291-CoreAndAdmin.patch [ 12597454 ]
          Junping Du made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 2.1.0-beta [ 12324029 ]
          Junping Du made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Junping Du made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Junping Du made changes -
          Summary Dynamic resource configuration [Umbrella] Dynamic resource configuration
          Junping Du made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Inigo Goiri made changes -
          Link This issue relates to YARN-3482 [ YARN-3482 ]

            People

            • Assignee:
              Junping Du
              Reporter:
              Junping Du
            • Votes:
              6 Vote for this issue
              Watchers:
              63 Start watching this issue

              Dates

              • Created:
                Updated:

                Development