Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1011

[Umbrella] Schedule containers based on utilization of currently allocated containers

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently RM allocates containers and assumes resources allocated are utilized.

      RM can, and should, get to a point where it measures utilization of allocated containers and, if appropriate, allocate more (speculative?) containers.

        Attachments

        1. patch-for-yarn-1011.patch
          62 kB
          Konstantinos Karanasos
        2. yarn-1011-design-v0.pdf
          44 kB
          Karthik Kambatla
        3. yarn-1011-design-v1.pdf
          48 kB
          Karthik Kambatla
        4. yarn-1011-design-v2.pdf
          66 kB
          Karthik Kambatla
        5. yarn-1011-design-v3.pdf
          90 kB
          Haibo Chen

        Issue Links

        is related to

        New Feature - A new feature of the product, which has yet to be developed. YARN-8849 DynoYARN: A simulation and testing infrastructure for YARN clusters

        • Major - Major loss of function.
        • Open
        1.
        		 FS should watch node resource utilization and allocate opportunistic containers if appropriate Sub-task Resolved Haibo Chen Actions
        2.
        		 Common scheduler changes to support scheduler-specific oversubscription implementations Sub-task Resolved Haibo Chen Actions
        3.
        		 Provide a knob to turn on over-allocation Sub-task Resolved Karthik Kambatla Actions
        4.
        		 Expose per-application over-allocation info in the Resource Manager Sub-task Resolved Haibo Chen Actions
        5.
        		 Add separate NM overallocation thresholds for cpu and memory Sub-task Resolved Haibo Chen Actions
        6.
        		 Add NM preemption of opportunistic containers when utilization goes high Sub-task Resolved Haibo Chen Actions
        7.
        		 Add NM support to launch opportunistic containers based on overallocation Sub-task Resolved Haibo Chen Actions
        8.
        		 Refactor ContainerScheduler to make oversubscription change easier Sub-task Resolved Haibo Chen Actions
        9.
        		 Add separate NM preemption thresholds for cpu and memory Sub-task Resolved Haibo Chen Actions
        10.
        		 Crash NM at start time if oversubscription is on but LinuxContainerExcutor or cgroup is off Sub-task Resolved Haibo Chen Actions
        11.
        		 Add a configuration to cap how much a NM can be overallocated Sub-task Resolved Haibo Chen Actions
        12.
        		 Fair Scheduler to explicitly promote OPPORTUNISITIC containers locally at the node where they're running Sub-task Resolved Haibo Chen Actions
        13.
        		 Allow resource request to opt out of oversubscription in Fair Scheduler Sub-task Resolved Haibo Chen Actions
        14.
        		 Expose per-node over-allocation info in Node Report Sub-task Resolved Haibo Chen Actions
        15.
        		 Preserve execution type for containers to be increased by AM post YARN-1015 Sub-task Resolved Haibo Chen Actions
        16.
        		 Exclude findbugs warnings on SchedulerNode.numGuaranteedContainers and numOpportunisticContainers Sub-task Resolved Haibo Chen Actions
        17.
        		 Make sure AM is not scheduled on an opportunistic container Sub-task Resolved Haibo Chen Actions
        18.
        		 Add queue-level control to allow all applications in a queue to opt-out Sub-task Resolved Haibo Chen Actions
        19.
        		 Don't start opportunistic containers at container scheduler/finish event with over-allocation Sub-task Resolved Haibo Chen Actions
        20.
        		 Use aggregate container utilization instead of node utilization to determine resources available for oversubscription Sub-task Resolved Haibo Chen Actions
        21.
        		 Refactor AbstractYarnScheduler and CapacityScheduler OPPORTUNISTIC container completion codepaths Sub-task Resolved Haibo Chen Actions
        22.
        		 FairScheduler crashes RM with oversubscription turned on if an application is killed. Sub-task Resolved Haibo Chen Actions
        23.
        		 Improve debug messages for NM preemption of OPPORTUNISTIC containers Sub-task Resolved Haibo Chen Actions
        24.
        		 Fair scheduler promotion does not update container execution type and token Sub-task Resolved Haibo Chen Actions
        25.
        		 Plumb aggregated application resource utilization from the NM to RM Sub-task Resolved Arun Suresh Actions
        26.
        		 Add documentation of oversubscription Sub-task Open Haibo Chen Actions
        27.
        		 CS should watch resource utilization of containers and allocate speculative containers if appropriate Sub-task Open Arun Suresh Actions
        28.
        		 Allow Applications to demand Guaranteed Containers with Capacity Scheduler Sub-task Open Arun Suresh Actions
        29.
        		 Handle enforceExecutionType == false in AMRMClient Sub-task Open Arun Suresh Actions
        30.
        		 SnapshotBasedOverAllocationPolicy always caps the amount of memory availabe to 4 GBs Sub-task Resolved Haibo Chen Actions
        31.
        		 ResourceUtilization cpu is misused in oversubscription as a percentage Sub-task Resolved Haibo Chen Actions
        32.
        		 Fair Scheduler promotion does not update container execution type when the application is killed Sub-task Open Unassigned Actions
        33.
        		 NM crashes because Fair scheduler promotes a container that has not been pulled by AM Sub-task Open Unassigned Actions

          Activity
          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              kasha Karthik Kambatla
              Reporter:
              acmurthy Arun Murthy

              Dates

              • Created:
                Updated:

                Issue deployment