Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1011

[Umbrella] Schedule containers based on utilization of currently allocated containers

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently RM allocates containers and assumes resources allocated are utilized.

      RM can, and should, get to a point where it measures utilization of allocated containers and, if appropriate, allocate more (speculative?) containers.

        Attachments

        1. yarn-1011-design-v3.pdf
          90 kB
          Haibo Chen
        2. yarn-1011-design-v2.pdf
          66 kB
          Karthik Kambatla
        3. yarn-1011-design-v1.pdf
          48 kB
          Karthik Kambatla
        4. yarn-1011-design-v0.pdf
          44 kB
          Karthik Kambatla
        5. patch-for-yarn-1011.patch
          62 kB
          Konstantinos Karanasos

          Issue Links

          1.
          Report NM aggregated container resource utilization in heartbeat Sub-task Resolved Íñigo Goiri  
          2.
          CS should watch resource utilization of containers and allocate speculative containers if appropriate Sub-task Open Weiwei Yang  
          3.
          FS should watch node resource utilization and allocate opportunistic containers if appropriate Sub-task Resolved Haibo Chen  
          4.
          Collect memory/cpu usage on the node Sub-task Resolved Íñigo Goiri

          0%

          Original Estimate - 336h
          Remaining Estimate - 336h
          5.
          ResourceUtilization should be a part of NodeReport API. Sub-task Resolved Naganarasimha G R  
          6.
          ResourceUtilization should be a part of NodeInfo REST API Sub-task Resolved Sunil Govindan  
          7.
          ResourceUtilization should be a part of yarn node CLI Sub-task Resolved Sunil Govindan  
          8.
          Plumb resource-utilization info in node heartbeat through to the scheduler Sub-task Resolved Íñigo Goiri  
          9.
          Common scheduler changes to support scheduler-specific oversubscription implementations Sub-task Resolved Haibo Chen  
          10.
          Provide a knob to turn on over-allocation Sub-task Resolved Karthik Kambatla  
          11.
          Handle enforceExecutionType == false in AMRMClient Sub-task Open Arun Suresh  
          12.
          Expose per-application over-allocation info in the Resource Manager Sub-task Resolved Haibo Chen  
          13.
          Add separate NM overallocation thresholds for cpu and memory Sub-task Resolved Haibo Chen  
          14.
          Add container type awareness in ResourceHandlers. Sub-task Resolved Miklos Szegedi  
          15.
          Use cgroup to get container resource utilization Sub-task Resolved Miklos Szegedi  
          16.
          Add NM preemption of opportunistic containers when utilization goes high Sub-task Resolved Haibo Chen  
          17.
          Add cpu cgroup configurations for opportunistic containers Sub-task Resolved Miklos Szegedi  
          18.
          Add memory cgroup settings for opportunistic containers Sub-task Resolved Miklos Szegedi  
          19.
          Add NM support to launch opportunistic containers based on overallocation Sub-task Resolved Haibo Chen  
          20.
          Consolidate NM overallocation thresholds with ResourceTypes Sub-task Open Haibo Chen  
          21.
          Refactor ContainerScheduler to make oversubscription change easier Sub-task Resolved Haibo Chen  
          22.
          Add separate NM preemption thresholds for cpu and memory Sub-task Resolved Haibo Chen  
          23.
          NM overallocation based on over-time rather than snapshot utilization Sub-task Open Haibo Chen  
          24.
          Crash NM at start time if oversubscription is on but LinuxContainerExcutor or cgroup is off Sub-task Resolved Haibo Chen  
          25.
          Add a configuration to cap how much a NM can be overallocated Sub-task Resolved Haibo Chen  
          26.
          Fair Scheduler to explicitly promote OPPORTUNISITIC containers locally at the node where they're running Sub-task Resolved Haibo Chen  
          27.
          Add unit test for NM to launch OPPORTUNISTIC container for overallocation Sub-task Resolved Haibo Chen  
          28.
          Add opportunity to start containers while periodically checking for preemption Sub-task Resolved Haibo Chen  
          29.
          Scheduler-agnostic RM changes support oversubscription Sub-task Resolved Haibo Chen  
          30.
          Add unit test for NM to preempt OPPORTUNISTIC containers under high utilization Sub-task Resolved Haibo Chen  
          31.
          Allow resource request to opt out of oversubscription in Fair Scheduler Sub-task Resolved Haibo Chen  
          32.
          Handle Container ExecType update (Promotion/Demotion) in cgroups resource handlers Sub-task Resolved Miklos Szegedi  
          33.
          Add documentation of oversubscription Sub-task Open Haibo Chen  
          34.
          Expose per-node over-allocation info in Node Report Sub-task Resolved Haibo Chen  
          35.
          Preserve execution type for containers to be increased by AM post YARN-1015 Sub-task Resolved Haibo Chen  
          36.
          Exclude findbugs warnings on SchedulerNode.numGuaranteedContainers and numOpportunisticContainers Sub-task Resolved Haibo Chen  
          37.
          Make sure AM is not scheduled on an opportunistic container Sub-task Resolved Haibo Chen  
          38.
          Add queue-level control to allow all applications in a queue to opt-out Sub-task Resolved Haibo Chen  
          39.
          Create another implementation of ContainerScheduler to support NM overallocation Sub-task Resolved Haibo Chen  
          40.
          Don't start opportunistic containers at container scheduler/finish event with over-allocation Sub-task Resolved Haibo Chen  
          41.
          Fair scheduler promotion does not update container execution type and token Sub-task Patch Available Haibo Chen  

            Activity

              People

              • Assignee:
                kasha Karthik Kambatla
                Reporter:
                acmurthy Arun C Murthy
              • Votes:
                7 Vote for this issue
                Watchers:
                98 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - 336h
                  336h
                  Remaining:
                  Remaining Estimate - 336h
                  336h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified