Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-291

[Umbrella] Dynamic resource configuration

    Details

      Description

      The current Hadoop YARN resource management logic assumes per node resource is static during the lifetime of the NM process. Allowing run-time configuration on per node resource will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist with other workloads on the same hardware efficiently, whether or not the environment is virtualized. More background and design details can be found in attached proposal.

        Attachments

        1. YARN-291-all-v1.patch
          66 kB
          Junping Du
        2. YARN-291-core-HeartBeatAndScheduler-01.patch
          31 kB
          Junping Du
        3. YARN-291-JMXInterfaceOnNM-02.patch
          4 kB
          Junping Du
        4. YARN-291-AddClientRMProtocolToSetNodeResource-03.patch
          25 kB
          Junping Du
        5. YARN-291-YARNClientCommandline-04.patch
          8 kB
          Junping Du
        6. YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch
          6 kB
          Junping Du
        7. Elastic Resources for YARN-v0.2.pdf
          512 kB
          Junping Du
        8. YARN-291-CoreAndAdmin.patch
          46 kB
          Junping Du

          Issue Links

          1.
          Dynamic node resource configuration: core scheduler changes Sub-task Closed Junping Du
          2.
          Add updateNodeResource in ResourceManagerAdministrationProtocol Sub-task Closed Junping Du
          3.
          Add Admin API for supporting node resource configuration in command line Sub-task Resolved Íñigo Goiri
          4.
          REST API support for node resource configuration Sub-task Open Tsuyoshi Ozawa
          5.
          JMX support for node resource configuration Sub-task Open Unassigned
          6.
          Keep NM resource updated through dynamic resource config for RM/NM restart Sub-task Resolved Junping Du
          7.
          In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task. Sub-task Open Junping Du
          8.
          Dynamic resource configuration feature can be configured to enable or disable and persistent on setting or not Sub-task Open Junping Du
          9.
          Replace set resource change on RMNode/SchedulerNode directly with event notification. Sub-task Resolved Junping Du
          10.
          Document Dynamic Resource Configuration feature Sub-task Open Junping Du
          11.
          Add admin ACL check to AdminService#updateNodeResource() Sub-task Resolved Kenji Kikushima
          12.
          ResouceOption's overcommitTimeout should be respected during resource update on NM Sub-task Open Junping Du
          13.
          Dynamic NM Resources Configuration file should be simplified. Sub-task Open Junping Du
          14.
          yarn rmadmin -updateNodeResource doesn't work Sub-task Resolved Junping Du
          15.
          updateNodeResource CLI should not accept negative values for resource Sub-task Resolved Junping Du
          16.
          RM Admin CLI for refreshNodesResources throws NPE when nothing is configured Sub-task Resolved Naganarasimha G R
          17.
          NM side resource value should get updated if change applied in RM side Sub-task Resolved Junping Du
          18.
          ContainersMonitorImpl need be adjusted when NM resource changed. Sub-task Open Junping Du

            Activity

              People

              • Assignee:
                djp Junping Du
                Reporter:
                djp Junping Du
              • Votes:
                9 Vote for this issue
                Watchers:
                75 Start watching this issue

                Dates

                • Created:
                  Updated: