Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1011

[Umbrella] Schedule containers based on utilization of currently allocated containers

          Details

          • Type: New Feature
          • Status: Open
          • Priority: Major
          • Resolution: Unresolved
          • Affects Version/s: None
          • Fix Version/s: None
          • Component/s: None
          • Labels:
            None

            Description

            Currently RM allocates containers and assumes resources allocated are utilized.

            RM can, and should, get to a point where it measures utilization of allocated containers and, if appropriate, allocate more (speculative?) containers.

              Attachments

              1. patch-for-yarn-1011.patch
                62 kB
                Konstantinos Karanasos
              2. yarn-1011-design-v0.pdf
                44 kB
                Karthik Kambatla
              3. yarn-1011-design-v1.pdf
                48 kB
                Karthik Kambatla
              4. yarn-1011-design-v2.pdf
                66 kB
                Karthik Kambatla
              5. yarn-1011-design-v3.pdf
                90 kB
                Haibo Chen

                Issue Links

                is related to

                New Feature - A new feature of the product, which has yet to be developed. YARN-8849 DynoYARN: A simulation and testing infrastructure for YARN clusters

                • Major - Major loss of function.
                • Open
                1.
                		 FS should watch node resource utilization and allocate opportunistic containers if appropriate Sub-task Resolved Haibo Chen
                2.
                		 Common scheduler changes to support scheduler-specific oversubscription implementations Sub-task Resolved Haibo Chen
                3.
                		 Provide a knob to turn on over-allocation Sub-task Resolved Karthik Kambatla
                4.
                		 Expose per-application over-allocation info in the Resource Manager Sub-task Resolved Haibo Chen
                5.
                		 Add separate NM overallocation thresholds for cpu and memory Sub-task Resolved Haibo Chen
                6.
                		 Add NM preemption of opportunistic containers when utilization goes high Sub-task Resolved Haibo Chen
                7.
                		 Add NM support to launch opportunistic containers based on overallocation Sub-task Resolved Haibo Chen
                8.
                		 Refactor ContainerScheduler to make oversubscription change easier Sub-task Resolved Haibo Chen
                9.
                		 Add separate NM preemption thresholds for cpu and memory Sub-task Resolved Haibo Chen
                10.
                		 Crash NM at start time if oversubscription is on but LinuxContainerExcutor or cgroup is off Sub-task Resolved Haibo Chen
                11.
                		 Add a configuration to cap how much a NM can be overallocated Sub-task Resolved Haibo Chen
                12.
                		 Fair Scheduler to explicitly promote OPPORTUNISITIC containers locally at the node where they're running Sub-task Resolved Haibo Chen
                13.
                		 Allow resource request to opt out of oversubscription in Fair Scheduler Sub-task Resolved Haibo Chen
                14.
                		 Expose per-node over-allocation info in Node Report Sub-task Resolved Haibo Chen
                15.
                		 Preserve execution type for containers to be increased by AM post YARN-1015 Sub-task Resolved Haibo Chen
                16.
                		 Exclude findbugs warnings on SchedulerNode.numGuaranteedContainers and numOpportunisticContainers Sub-task Resolved Haibo Chen
                17.
                		 Make sure AM is not scheduled on an opportunistic container Sub-task Resolved Haibo Chen
                18.
                		 Add queue-level control to allow all applications in a queue to opt-out Sub-task Resolved Haibo Chen
                19.
                		 Don't start opportunistic containers at container scheduler/finish event with over-allocation Sub-task Resolved Haibo Chen
                20.
                		 Use aggregate container utilization instead of node utilization to determine resources available for oversubscription Sub-task Resolved Haibo Chen
                21.
                		 Refactor AbstractYarnScheduler and CapacityScheduler OPPORTUNISTIC container completion codepaths Sub-task Resolved Haibo Chen
                22.
                		 FairScheduler crashes RM with oversubscription turned on if an application is killed. Sub-task Resolved Haibo Chen
                23.
                		 Improve debug messages for NM preemption of OPPORTUNISTIC containers Sub-task Resolved Haibo Chen
                24.
                		 Fair scheduler promotion does not update container execution type and token Sub-task Resolved Haibo Chen
                25.
                		 Plumb aggregated application resource utilization from the NM to RM Sub-task Resolved Arun Suresh
                26.
                		 Add documentation of oversubscription Sub-task Open Haibo Chen
                27.
                		 CS should watch resource utilization of containers and allocate speculative containers if appropriate Sub-task Open Arun Suresh
                28.
                		 Allow Applications to demand Guaranteed Containers with Capacity Scheduler Sub-task Open Arun Suresh
                29.
                		 Handle enforceExecutionType == false in AMRMClient Sub-task Open Arun Suresh
                30.
                		 SnapshotBasedOverAllocationPolicy always caps the amount of memory availabe to 4 GBs Sub-task Resolved Haibo Chen
                31.
                		 ResourceUtilization cpu is misused in oversubscription as a percentage Sub-task Resolved Haibo Chen
                32.
                		 Fair Scheduler promotion does not update container execution type when the application is killed Sub-task Open Unassigned
                33.
                		 NM crashes because Fair scheduler promotes a container that has not been pulled by AM Sub-task Open Unassigned

                  Activity

                    People

                    • Assignee:
                      kasha Karthik Kambatla
                      Reporter:
                      acmurthy Arun C Murthy
                    • Votes:
                      11 Vote for this issue
                      Watchers:
                      109 Start watching this issue

                      Dates

                      • Created:
                        Updated: