Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4597

Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle

    Details

      Description

      Currently, the NM immediately launches containers after resource localization. Several features could be more cleanly implemented if the NM included a separate stage for reserving resources.

      1. YARN-4597.001.patch
        159 kB
        Arun Suresh
      2. YARN-4597.002.patch
        180 kB
        Arun Suresh
      3. YARN-4597.003.patch
        190 kB
        Arun Suresh
      4. YARN-4597.004.patch
        195 kB
        Arun Suresh
      5. YARN-4597.005.patch
        199 kB
        Arun Suresh
      6. YARN-4597.006.patch
        208 kB
        Arun Suresh
      7. YARN-4597.007.patch
        206 kB
        Arun Suresh
      8. YARN-4597.008.patch
        209 kB
        Arun Suresh
      9. YARN-4597.009.patch
        209 kB
        Arun Suresh
      10. YARN-4597.010.patch
        220 kB
        Arun Suresh
      11. YARN-4597.011.patch
        224 kB
        Arun Suresh
      12. YARN-4597.012.patch
        227 kB
        Arun Suresh
      13. YARN-4597.013.patch
        229 kB
        Arun Suresh

        Issue Links

          Activity

          Hide
          chris.douglas Chris Douglas added a comment -

          The ContainerLaunchContext (CLC) specifies the prerequisites for starting a container on a node. These include setting up user/application directories and downloading dependencies to the NM cache (localization). The NM assumes that an authenticated startContainer request has not overbooked resources on the node, so resources are only reserved/enforced during the container launch and execution.

          This JIRA proposes to add a phase between localization and container launch to manage a collection of runnable containers. Similar to the localizer stage, a container will launch only after all the resources from its CLC are assigned by a local scheduler. The local scheduler will select containers to run based on priority, declared requirements, and by monitoring utilization on the node (YARN-1011).

          A few future and in-progress features motiviate this change.

          Preemption Instead of sending a kill when the RM selects a victim container, it could instead convert it from a GUARANTEED to an OPTIMISTIC container (YARN-4335). This has two benefits. First, the downgraded container can continue to run until a guaranteed container arrives and finishes localizing its dependencies, so the downgraded container has an opportunity to complete or checkpoint. When the guaranteed container moves from LOCALIZED to SCHEDULING, the local scheduler may select the victim (formerly guaranteed) container to be killed. [1] Second, the NM may elect to kill the victim container to run different optimistic containers, particularly short-running tasks.

          Optimistic scheduling and overprovisioning To support distributed scheduling (YARN-2877) and resource-aware scheduling (YARN-1011), the NM needs a component to select containers that are ready to run. The local scheduler can not only select tasks to run based on monitoring, it can also make offers to running containers using durations attached to leases [2]. Based on recent observations, it may start containers that oversubscribe the node, or delay starting containers if a lease is close to expiring (i.e., the container is likely to complete).

          Long-running services. Note that by separating the local scheduler, both that module and the localizer could be opened up as services provided by the NM. The localizer could also be extended to prioritize downloads among OPTIMISTIC containers (possibly preemptable by GUARANTEED, and to group containers based on their dependencies (e.g., avoid downloading a large dep for fewer than N optimistic containers). By exposing these services, the NM can assist with the following:

          1. Resource spikes. If a service container needs to spike temporarily, it may not need guaranteed resources (YARN-1197). Containers requiring low-latency elasticity could request optimistic resources instead of peak provisioning, resizing, or using workarounds like Llama. If the local scheduler is addressable by local containers, then the lease could be logical (i.e., not start a process). Resources assigned to a RUNNING container could be published rather than triggering a launch. One could also imagine service workers marking some resources as unused, while retaining the authority to spike into them ("subleasing" them to opportunistic containers) by reclaiming them through the local scheduler.
          2. Upgrades. If the container needs to pull new dependencies, it could use the NM Localizer rather of coordinating the download itself.
          3. Maintenance tasks. Services often need to clean up, compact, scrub, and checkpoint local data. Right now, each service needs to independnetly monitor resource utilization to back off saturated resources (particularly disks). Coordination between services is difficult. In contrast, one could schedule tasks like block scrubbing as optimistic tasks in the NM to avoid interrupting services that are spiking. This is similar in spirit to distributed scheduling insofar as it does not involve the RM and targets a single host (i.e., the host the container is running on).

          [1] Though it was selected as a victim by the RM, the local scheduler may decide to kill a different OPTIMISTIC container when the guaranteed container requests resources. For example, if a container completes on the node after the RM selected the victim, then the NM may elect to kill a smaller optimistic process if it is sufficient to satisfy the guarantee.
          [2] Discussion on duration in YARN-1039 was part of a broader conversation on support for long-running services (YARN-896).

          Show
          chris.douglas Chris Douglas added a comment - The ContainerLaunchContext (CLC) specifies the prerequisites for starting a container on a node. These include setting up user/application directories and downloading dependencies to the NM cache (localization). The NM assumes that an authenticated startContainer request has not overbooked resources on the node, so resources are only reserved/enforced during the container launch and execution. This JIRA proposes to add a phase between localization and container launch to manage a collection of runnable containers. Similar to the localizer stage, a container will launch only after all the resources from its CLC are assigned by a local scheduler . The local scheduler will select containers to run based on priority, declared requirements, and by monitoring utilization on the node ( YARN-1011 ). A few future and in-progress features motiviate this change. Preemption Instead of sending a kill when the RM selects a victim container, it could instead convert it from a GUARANTEED to an OPTIMISTIC container ( YARN-4335 ). This has two benefits. First, the downgraded container can continue to run until a guaranteed container arrives and finishes localizing its dependencies, so the downgraded container has an opportunity to complete or checkpoint. When the guaranteed container moves from LOCALIZED to SCHEDULING , the local scheduler may select the victim (formerly guaranteed) container to be killed. [1] Second, the NM may elect to kill the victim container to run different optimistic containers, particularly short-running tasks. Optimistic scheduling and overprovisioning To support distributed scheduling ( YARN-2877 ) and resource-aware scheduling ( YARN-1011 ), the NM needs a component to select containers that are ready to run. The local scheduler can not only select tasks to run based on monitoring, it can also make offers to running containers using durations attached to leases [2]. Based on recent observations, it may start containers that oversubscribe the node, or delay starting containers if a lease is close to expiring (i.e., the container is likely to complete). Long-running services . Note that by separating the local scheduler, both that module and the localizer could be opened up as services provided by the NM. The localizer could also be extended to prioritize downloads among OPTIMISTIC containers (possibly preemptable by GUARANTEED , and to group containers based on their dependencies (e.g., avoid downloading a large dep for fewer than N optimistic containers). By exposing these services, the NM can assist with the following: Resource spikes. If a service container needs to spike temporarily, it may not need guaranteed resources ( YARN-1197 ). Containers requiring low-latency elasticity could request optimistic resources instead of peak provisioning, resizing, or using workarounds like Llama . If the local scheduler is addressable by local containers, then the lease could be logical (i.e., not start a process). Resources assigned to a RUNNING container could be published rather than triggering a launch. One could also imagine service workers marking some resources as unused, while retaining the authority to spike into them ("subleasing" them to opportunistic containers) by reclaiming them through the local scheduler. Upgrades. If the container needs to pull new dependencies, it could use the NM Localizer rather of coordinating the download itself. Maintenance tasks. Services often need to clean up, compact, scrub, and checkpoint local data. Right now, each service needs to independnetly monitor resource utilization to back off saturated resources (particularly disks). Coordination between services is difficult. In contrast, one could schedule tasks like block scrubbing as optimistic tasks in the NM to avoid interrupting services that are spiking. This is similar in spirit to distributed scheduling insofar as it does not involve the RM and targets a single host (i.e., the host the container is running on). [1] Though it was selected as a victim by the RM, the local scheduler may decide to kill a different OPTIMISTIC container when the guaranteed container requests resources. For example, if a container completes on the node after the RM selected the victim, then the NM may elect to kill a smaller optimistic process if it is sufficient to satisfy the guarantee. [2] Discussion on duration in YARN-1039 was part of a broader conversation on support for long-running services ( YARN-896 ).
          Hide
          asuresh Arun Suresh added a comment -

          +1 to the idea Chris Douglas. Agree that it would simplify some design choices we make for modification to the ContianersMonitor and ContainerManager side for YARN-2877. Would love to help out with this (I'd even volunteer to take a first crack at it since it directly impacts our Distributed Scheduling work)

          Show
          asuresh Arun Suresh added a comment - +1 to the idea Chris Douglas . Agree that it would simplify some design choices we make for modification to the ContianersMonitor and ContainerManager side for YARN-2877 . Would love to help out with this (I'd even volunteer to take a first crack at it since it directly impacts our Distributed Scheduling work)
          Hide
          chris.douglas Chris Douglas added a comment - - edited

          Thanks, Arun. Please feel free to take this over. It's only justified in context with these other changes.

          Show
          chris.douglas Chris Douglas added a comment - - edited Thanks, Arun. Please feel free to take this over. It's only justified in context with these other changes.
          Hide
          kasha Karthik Kambatla added a comment -

          Makes a lot of sense. Thanks for filing this, Chris.

          Show
          kasha Karthik Kambatla added a comment - Makes a lot of sense. Thanks for filing this, Chris.
          Hide
          asuresh Arun Suresh added a comment -

          Based on discussions with Karthik Kambatla, Konstantinos Karanasos, Subru Krishnan..
          I propose the following:

          1. Rename SCHEDULE (as proposed by this JIRA) to QUEUED
          2. Move the management of Queued container from QueuingContainerManagerImp into the Local Scheduler.
          Show
          asuresh Arun Suresh added a comment - Based on discussions with Karthik Kambatla , Konstantinos Karanasos , Subru Krishnan .. I propose the following: Rename SCHEDULE (as proposed by this JIRA) to QUEUED Move the management of Queued container from QueuingContainerManagerImp into the Local Scheduler.
          Hide
          asuresh Arun Suresh added a comment -

          Attaching initial patch.

          • Introduced a ContainerScheduler : The Scheduler can decide when a Container can be launched. This can be done based on policy. Currently, there is flag that it checks to see if Queuing is enabled, it performs the role of the QueuingContainerManagerImpl
          • Removed the QueuingContainerManagerImpl : As described above, it is not required anymore. Most of the logic has been moved to the Scheduler.
          • There is no need for a QueuingContext anymore.
          • TestQueuingContainerManager has been moved to
            TestContainerSchedulerQueuing
          • The LOCALIZED container state has been renamed to SCHEDULED

          Since a lot of code paths have been simplified, a lot of code changes are actually deletions.

          Do take a look. (cc: Karthik Kambatla, Chris Douglas, Varun Vasudev, Jian He, Konstantinos Karanasos, Subru Krishnan)

          Show
          asuresh Arun Suresh added a comment - Attaching initial patch. Introduced a ContainerScheduler : The Scheduler can decide when a Container can be launched. This can be done based on policy. Currently, there is flag that it checks to see if Queuing is enabled, it performs the role of the QueuingContainerManagerImpl Removed the QueuingContainerManagerImpl : As described above, it is not required anymore. Most of the logic has been moved to the Scheduler. There is no need for a QueuingContext anymore. TestQueuingContainerManager has been moved to TestContainerSchedulerQueuing The LOCALIZED container state has been renamed to SCHEDULED Since a lot of code paths have been simplified, a lot of code changes are actually deletions. Do take a look. (cc: Karthik Kambatla , Chris Douglas , Varun Vasudev , Jian He , Konstantinos Karanasos , Subru Krishnan )
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 10 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 6m 55s trunk passed
          +1 compile 2m 25s trunk passed
          +1 checkstyle 0m 45s trunk passed
          +1 mvnsite 1m 43s trunk passed
          +1 mvneclipse 0m 58s trunk passed
          +1 findbugs 2m 39s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 21s the patch passed
          +1 compile 2m 17s the patch passed
          -1 javac 2m 17s hadoop-yarn-project_hadoop-yarn generated 3 new + 32 unchanged - 3 fixed = 35 total (was 35)
          -1 checkstyle 0m 42s hadoop-yarn-project/hadoop-yarn: The patch generated 28 new + 480 unchanged - 9 fixed = 508 total (was 489)
          +1 mvnsite 1m 34s the patch passed
          +1 mvneclipse 0m 49s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 16s hadoop-yarn-api in the patch passed.
          +1 javadoc 0m 15s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 239 unchanged - 1 fixed = 239 total (was 240)
          +1 javadoc 0m 10s hadoop-yarn-server-tests in the patch passed.
          +1 javadoc 0m 12s hadoop-yarn-client in the patch passed.
          +1 unit 0m 23s hadoop-yarn-api in the patch passed.
          -1 unit 12m 15s hadoop-yarn-server-nodemanager in the patch failed.
          -1 unit 4m 25s hadoop-yarn-server-tests in the patch failed.
          -1 unit 18m 52s hadoop-yarn-client in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          64m 58s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
            Unused field:NodeManager.java
          Failed junit tests hadoop.yarn.server.nodemanager.TestNodeStatusUpdater
            hadoop.yarn.server.nodemanager.containermanager.TestContainerManagerRecovery
            hadoop.yarn.server.nodemanager.containermanager.container.TestContainer
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation
            hadoop.yarn.client.api.impl.TestAMRMClient
            hadoop.yarn.client.api.impl.TestNMClient



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831552/YARN-4597.001.patch
          JIRA Issue YARN-4597
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux fd0e08be5b94 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ef7f06f
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13278/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13278/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 10 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 55s trunk passed +1 compile 2m 25s trunk passed +1 checkstyle 0m 45s trunk passed +1 mvnsite 1m 43s trunk passed +1 mvneclipse 0m 58s trunk passed +1 findbugs 2m 39s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 21s the patch passed +1 compile 2m 17s the patch passed -1 javac 2m 17s hadoop-yarn-project_hadoop-yarn generated 3 new + 32 unchanged - 3 fixed = 35 total (was 35) -1 checkstyle 0m 42s hadoop-yarn-project/hadoop-yarn: The patch generated 28 new + 480 unchanged - 9 fixed = 508 total (was 489) +1 mvnsite 1m 34s the patch passed +1 mvneclipse 0m 49s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 16s hadoop-yarn-api in the patch passed. +1 javadoc 0m 15s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 239 unchanged - 1 fixed = 239 total (was 240) +1 javadoc 0m 10s hadoop-yarn-server-tests in the patch passed. +1 javadoc 0m 12s hadoop-yarn-client in the patch passed. +1 unit 0m 23s hadoop-yarn-api in the patch passed. -1 unit 12m 15s hadoop-yarn-server-nodemanager in the patch failed. -1 unit 4m 25s hadoop-yarn-server-tests in the patch failed. -1 unit 18m 52s hadoop-yarn-client in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 64m 58s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager   Unused field:NodeManager.java Failed junit tests hadoop.yarn.server.nodemanager.TestNodeStatusUpdater   hadoop.yarn.server.nodemanager.containermanager.TestContainerManagerRecovery   hadoop.yarn.server.nodemanager.containermanager.container.TestContainer   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation   hadoop.yarn.client.api.impl.TestAMRMClient   hadoop.yarn.client.api.impl.TestNMClient Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831552/YARN-4597.001.patch JIRA Issue YARN-4597 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fd0e08be5b94 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ef7f06f Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt https://builds.apache.org/job/PreCommit-YARN-Build/13278/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13278/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13278/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment -

          Updating patch.

          • Simplifying ContainerScheduler code further : No need for synchronized data structures, since the events are guaranteed to be ordered.
          • Fixed most of the testcases. The TestContainer is a bit more involved.. will include that in the next patch.
          • As per Karthik Kambatla's suggestion, instead of using a boolean to signify if queuing should be enabled, the configuration has been replaced with 'max-queue-length' so signify the size of the queue. A default value of 0 implies queuing is disabled.
          Show
          asuresh Arun Suresh added a comment - Updating patch. Simplifying ContainerScheduler code further : No need for synchronized data structures, since the events are guaranteed to be ordered. Fixed most of the testcases. The TestContainer is a bit more involved.. will include that in the next patch. As per Karthik Kambatla 's suggestion, instead of using a boolean to signify if queuing should be enabled, the configuration has been replaced with 'max-queue-length' so signify the size of the queue. A default value of 0 implies queuing is disabled.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 15 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 9m 11s trunk passed
          +1 compile 9m 10s trunk passed
          +1 checkstyle 2m 4s trunk passed
          +1 mvnsite 3m 41s trunk passed
          +1 mvneclipse 1m 51s trunk passed
          +1 findbugs 4m 50s trunk passed
          +1 javadoc 1m 58s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 2m 44s the patch passed
          +1 compile 9m 1s the patch passed
          +1 cc 9m 1s the patch passed
          -1 javac 9m 1s root generated 3 new + 705 unchanged - 3 fixed = 708 total (was 708)
          -1 checkstyle 2m 4s root: The patch generated 35 new + 782 unchanged - 12 fixed = 817 total (was 794)
          +1 mvnsite 3m 33s the patch passed
          +1 mvneclipse 1m 52s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 0m 58s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 17s hadoop-yarn-api in the patch passed.
          +1 javadoc 0m 21s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 239 unchanged - 1 fixed = 239 total (was 240)
          +1 javadoc 0m 22s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 javadoc 0m 12s hadoop-yarn-server-tests in the patch passed.
          +1 javadoc 0m 15s hadoop-yarn-client in the patch passed.
          +1 javadoc 0m 13s hadoop-mapreduce-client-jobclient in the patch passed.
          -1 unit 0m 27s hadoop-yarn-api in the patch failed.
          -1 unit 12m 34s hadoop-yarn-server-nodemanager in the patch failed.
          +1 unit 38m 40s hadoop-yarn-server-resourcemanager in the patch passed.
          -1 unit 4m 26s hadoop-yarn-server-tests in the patch failed.
          -1 unit 15m 55s hadoop-yarn-client in the patch failed.
          -1 unit 115m 34s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          249m 22s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
            Unused field:NodeManager.java
          Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields
            hadoop.yarn.server.nodemanager.TestNodeManagerResync
            hadoop.yarn.server.nodemanager.containermanager.container.TestContainer
            hadoop.yarn.server.nodemanager.TestEventFlow
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.client.api.impl.TestNMClient
          Timed out junit tests org.apache.hadoop.mapred.TestMROpportunisticMaps



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832373/YARN-4597.002.patch
          JIRA Issue YARN-4597
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 4fe72bb56361 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bea004e
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13328/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13328/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 15 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 9m 11s trunk passed +1 compile 9m 10s trunk passed +1 checkstyle 2m 4s trunk passed +1 mvnsite 3m 41s trunk passed +1 mvneclipse 1m 51s trunk passed +1 findbugs 4m 50s trunk passed +1 javadoc 1m 58s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 2m 44s the patch passed +1 compile 9m 1s the patch passed +1 cc 9m 1s the patch passed -1 javac 9m 1s root generated 3 new + 705 unchanged - 3 fixed = 708 total (was 708) -1 checkstyle 2m 4s root: The patch generated 35 new + 782 unchanged - 12 fixed = 817 total (was 794) +1 mvnsite 3m 33s the patch passed +1 mvneclipse 1m 52s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 58s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 17s hadoop-yarn-api in the patch passed. +1 javadoc 0m 21s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 239 unchanged - 1 fixed = 239 total (was 240) +1 javadoc 0m 22s hadoop-yarn-server-resourcemanager in the patch passed. +1 javadoc 0m 12s hadoop-yarn-server-tests in the patch passed. +1 javadoc 0m 15s hadoop-yarn-client in the patch passed. +1 javadoc 0m 13s hadoop-mapreduce-client-jobclient in the patch passed. -1 unit 0m 27s hadoop-yarn-api in the patch failed. -1 unit 12m 34s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 38m 40s hadoop-yarn-server-resourcemanager in the patch passed. -1 unit 4m 26s hadoop-yarn-server-tests in the patch failed. -1 unit 15m 55s hadoop-yarn-client in the patch failed. -1 unit 115m 34s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 249m 22s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager   Unused field:NodeManager.java Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields   hadoop.yarn.server.nodemanager.TestNodeManagerResync   hadoop.yarn.server.nodemanager.containermanager.container.TestContainer   hadoop.yarn.server.nodemanager.TestEventFlow   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.client.api.impl.TestNMClient Timed out junit tests org.apache.hadoop.mapred.TestMROpportunisticMaps Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832373/YARN-4597.002.patch JIRA Issue YARN-4597 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 4fe72bb56361 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bea004e Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt https://builds.apache.org/job/PreCommit-YARN-Build/13328/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13328/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13328/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          Glanced through the patch, few comments:

          • Wondering why KillWhileExitingTransition is added: It will override the exitcode of the container. I feel in this case we should preserve the previous exitcode, because the container failed/succeeded first before the kill event. Or maybe we don't even need this transition, as the behavior is that the kill is ignored.
          • NodeManager#containerScheduler, NMContext#containerScheduler variable is not used, we can remove.
          • I think it's fine to not expose the ContainerState#SCHEDULED state to the user, this state is mostly internal-facing and transient. User may not care which intermediate state the container is at.
                case SCHEDULED:
                  return org.apache.hadoop.yarn.api.records.ContainerState.SCHEDULED;
            
          Show
          jianhe Jian He added a comment - Glanced through the patch, few comments: Wondering why KillWhileExitingTransition is added: It will override the exitcode of the container. I feel in this case we should preserve the previous exitcode, because the container failed/succeeded first before the kill event. Or maybe we don't even need this transition, as the behavior is that the kill is ignored. NodeManager#containerScheduler, NMContext#containerScheduler variable is not used, we can remove. I think it's fine to not expose the ContainerState#SCHEDULED state to the user, this state is mostly internal-facing and transient. User may not care which intermediate state the container is at. case SCHEDULED: return org.apache.hadoop.yarn.api.records.ContainerState.SCHEDULED;
          Hide
          asuresh Arun Suresh added a comment - - edited

          Thanks for taking a look Jian He,

          Wondering why KillWhileExitingTransition is added..

          I had put it in there for debugging something... Left it there since I thought it was harmless... but, yeah looks like it does over-ride the exitcode. Will remove it. Good catch.

          • w.r.t ContainerState#SCHEDULED : Actually, I think we should expose this. We currently club NEW, LOCALIZING, LOCALIZED etc. into RUNNING, but the container is actually not running, and is thus misleading. SCHEDULED implies that some of the containers dependencies (resources for localization + some internal queuing/scheduling policy) have not yet been met.
            Prior to this, YARN-2877 had introduced the QUEUED return state. This would be visible to applications, if Queuing was enabled. This patch technically just renames QUEUED to SCHEDULED. Also, all containers will go thru the SCHEDULED state, not just the opportunistic ones (although, for guaranteed containers this will just be a pass-thru state)

          Another thing I was hoping for some input was, currently, the ContainerScheduler runs in the same thread as the ContainerManager's AsyncDispatcher started by the ContainerManager. Also, the Scheduler is triggered only by events. I was wondering if there is any merit pushing these events into a blocking queue as they arrive and have a separate thread take care of them. This will preserve the serial nature of operation (and thereby keep the code simple by not needing synchronized collections) and will not hold up the dispatcher from delivering other events while the scheduler is scheduling.
          A minor disadvantage, is that the NM will probably consume a thread that for the most part will be blocked on the queue. This thread could be used by one of the containers.

          Show
          asuresh Arun Suresh added a comment - - edited Thanks for taking a look Jian He , Wondering why KillWhileExitingTransition is added.. I had put it in there for debugging something... Left it there since I thought it was harmless... but, yeah looks like it does over-ride the exitcode. Will remove it. Good catch. w.r.t ContainerState#SCHEDULED : Actually, I think we should expose this. We currently club NEW, LOCALIZING, LOCALIZED etc. into RUNNING, but the container is actually not running, and is thus misleading. SCHEDULED implies that some of the containers dependencies (resources for localization + some internal queuing/scheduling policy) have not yet been met. Prior to this, YARN-2877 had introduced the QUEUED return state. This would be visible to applications, if Queuing was enabled. This patch technically just renames QUEUED to SCHEDULED. Also, all containers will go thru the SCHEDULED state, not just the opportunistic ones (although, for guaranteed containers this will just be a pass-thru state) Another thing I was hoping for some input was, currently, the ContainerScheduler runs in the same thread as the ContainerManager's AsyncDispatcher started by the ContainerManager. Also, the Scheduler is triggered only by events. I was wondering if there is any merit pushing these events into a blocking queue as they arrive and have a separate thread take care of them. This will preserve the serial nature of operation (and thereby keep the code simple by not needing synchronized collections) and will not hold up the dispatcher from delivering other events while the scheduler is scheduling. A minor disadvantage, is that the NM will probably consume a thread that for the most part will be blocked on the queue. This thread could be used by one of the containers.
          Hide
          jianhe Jian He added a comment -

          This will preserve the serial nature of operation (and thereby keep the code simple by not needing synchronized collections)

          I don't actually see a synchronized collection in the ContainerScheduler. I agree with the point that this could avoid holding up the main containerManager thread. we can have ContainerScheduler create its own dispatcher.

          Show
          jianhe Jian He added a comment - This will preserve the serial nature of operation (and thereby keep the code simple by not needing synchronized collections) I don't actually see a synchronized collection in the ContainerScheduler. I agree with the point that this could avoid holding up the main containerManager thread. we can have ContainerScheduler create its own dispatcher.
          Hide
          asuresh Arun Suresh added a comment -

          What i meant was, the current patch does not need synchronized collections ( unlike the QueuingContainerManager ), since it runs on the same thread as the ContainerManager's AsyncDispatcher.
          I will update the patch with using a linkedblockingqueue shortly.

          Show
          asuresh Arun Suresh added a comment - What i meant was, the current patch does not need synchronized collections ( unlike the QueuingContainerManager ), since it runs on the same thread as the ContainerManager's AsyncDispatcher. I will update the patch with using a linkedblockingqueue shortly.
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for working on this, Arun Suresh. The approach looks reasonable to me. The patch is pretty big; it might be easier to use Github PR or RB for a thorough review - especially for minor comments.

          High-level comments on the patch, keeping YARN-1011 work in mind:

          1. Really like that we are getting rid of QueuingContainer* classes, letting both guaranteed/queued containers go through the same code path
          2. In ContainerScheduler, I see there are two code paths leading to starting a container for when enough resources are available or not. Did you consider a single path where we queue containers directly and let another thread launch them? This thread could be triggered immediately on queuing a container, on completion of a container, and periodically for cases where we do resource oversubscription.
          3. The methods for killing containers as needed all seem to be hardcoded to only consider allocated resources. Can we abstract it out further to allow for passing either allocation or utilization based on whether oversubscription is enabled.
          4. Relatively minor: resourcesToFreeUp is initialized to container allocation on the node. Shouldn't it be initialized to zero? May be I am missing something.
          Show
          kasha Karthik Kambatla added a comment - Thanks for working on this, Arun Suresh . The approach looks reasonable to me. The patch is pretty big; it might be easier to use Github PR or RB for a thorough review - especially for minor comments. High-level comments on the patch, keeping YARN-1011 work in mind: Really like that we are getting rid of QueuingContainer* classes, letting both guaranteed/queued containers go through the same code path In ContainerScheduler, I see there are two code paths leading to starting a container for when enough resources are available or not. Did you consider a single path where we queue containers directly and let another thread launch them? This thread could be triggered immediately on queuing a container, on completion of a container, and periodically for cases where we do resource oversubscription. The methods for killing containers as needed all seem to be hardcoded to only consider allocated resources. Can we abstract it out further to allow for passing either allocation or utilization based on whether oversubscription is enabled. Relatively minor: resourcesToFreeUp is initialized to container allocation on the node. Shouldn't it be initialized to zero? May be I am missing something.
          Hide
          vvasudev Varun Vasudev added a comment -

          Thanks for the patch Arun Suresh.

          1)
          I agree with Karthik Kambatla on

          The methods for killing containers as needed all seem to be hardcoded to only consider allocated resources. Can we abstract it out further to allow for passing either allocation or utilization based on whether oversubscription is enabled.


          At some point, people will want to be able to plug in policies to decide which containers to kill. However, I wouldn't hold up the patch for it.

          2)
          The changes to BaseContainerManagerTest.java seem unnecessary.

          3)
          Can you explain why we need the synchronized block here -

           +    synchronized (this.containersAllocation) 
          Show
          vvasudev Varun Vasudev added a comment - Thanks for the patch Arun Suresh . 1) I agree with Karthik Kambatla on The methods for killing containers as needed all seem to be hardcoded to only consider allocated resources. Can we abstract it out further to allow for passing either allocation or utilization based on whether oversubscription is enabled. At some point, people will want to be able to plug in policies to decide which containers to kill. However, I wouldn't hold up the patch for it. 2) The changes to BaseContainerManagerTest.java seem unnecessary. 3) Can you explain why we need the synchronized block here - + synchronized ( this .containersAllocation)
          Hide
          asuresh Arun Suresh added a comment -

          Updating patch based on the thoughtful reviews from Varun Vasudev, Karthik Kambatla and Jian He ..

          The methods for killing containers as needed all seem to be hardcoded to only consider allocated resources. Can we abstract it out further to allow for passing either allocation or utilization based on whether oversubscription is enabled

          In the latest patch, I've introduced a ResourceUtilizationManager, the default implementation accumulates the allocated container resources. This can be made pluggable once we have resource utilization.

          Can you explain why we need the synchronized block here -

           +    synchronized (this.containersAllocation) 

          Aah.. its not required, removed it since access to this class is actually serialized.

          resourcesToFreeUp is initialized to container allocation on the node. Shouldn't it be initialized to zero? May be I am missing something

          The algorithm is as follows, assuming an NM has 10 slots, they are currently full (7 guaranteed and 3 opportunistic), and assume an incoming Guar container request:

          1. Start with currently utilized/allocated slots : 10
          2. Increase value of 1) with any guaranteed containers yet to start : 10 + 1(say 1 guaranteed request had come in before this) = 11
          3. Subtract from 2) total resources available to all containers : 11 - 10 = 1
          4. From 3) keep subtracting resources of all running Opportunistic containers (which have not already been marked to kill, in reverse startup order) till the value goes < 0.
          5. Kill all opportunistic containers identified in 4.

          I see there are two code paths leading to starting a container for when enough resources are available or not. Did you consider a single path where we queue containers directly and let another thread launch them.

          Good point. I did think about that, but that would introduce having the logic in a dedicated thread, which can be used to run containers. I was thinking we keep it as it is right now till a point when we really warrant that complexity.

          The tests (except TestContainer) all seem to run fine locally.

          Show
          asuresh Arun Suresh added a comment - Updating patch based on the thoughtful reviews from Varun Vasudev , Karthik Kambatla and Jian He .. The methods for killing containers as needed all seem to be hardcoded to only consider allocated resources. Can we abstract it out further to allow for passing either allocation or utilization based on whether oversubscription is enabled In the latest patch, I've introduced a ResourceUtilizationManager , the default implementation accumulates the allocated container resources. This can be made pluggable once we have resource utilization. Can you explain why we need the synchronized block here - + synchronized ( this .containersAllocation) Aah.. its not required, removed it since access to this class is actually serialized. resourcesToFreeUp is initialized to container allocation on the node. Shouldn't it be initialized to zero? May be I am missing something The algorithm is as follows, assuming an NM has 10 slots, they are currently full (7 guaranteed and 3 opportunistic), and assume an incoming Guar container request: Start with currently utilized/allocated slots : 10 Increase value of 1) with any guaranteed containers yet to start : 10 + 1(say 1 guaranteed request had come in before this) = 11 Subtract from 2) total resources available to all containers : 11 - 10 = 1 From 3) keep subtracting resources of all running Opportunistic containers (which have not already been marked to kill, in reverse startup order) till the value goes < 0. Kill all opportunistic containers identified in 4. I see there are two code paths leading to starting a container for when enough resources are available or not. Did you consider a single path where we queue containers directly and let another thread launch them. Good point. I did think about that, but that would introduce having the logic in a dedicated thread, which can be used to run containers. I was thinking we keep it as it is right now till a point when we really warrant that complexity. The tests (except TestContainer ) all seem to run fine locally.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user xslogic opened a pull request:

          https://github.com/apache/hadoop/pull/143

          YARN-4597: initial commit

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/xslogic/hadoop YARN-4597

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/143.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #143


          commit 44f42805c78a2a605889e5ff757a9996525b99e5
          Author: Arun Suresh <asuresh@apache.org>
          Date: 2016-10-18T01:49:53Z

          YARN-4597: initial commit


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user xslogic opened a pull request: https://github.com/apache/hadoop/pull/143 YARN-4597 : initial commit You can merge this pull request into a Git repository by running: $ git pull https://github.com/xslogic/hadoop YARN-4597 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/143.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #143 commit 44f42805c78a2a605889e5ff757a9996525b99e5 Author: Arun Suresh <asuresh@apache.org> Date: 2016-10-18T01:49:53Z YARN-4597 : initial commit
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 19 new or modified test files.
          0 mvndep 0m 42s Maven dependency ordering for branch
          +1 mvninstall 7m 21s trunk passed
          +1 compile 9m 14s trunk passed
          +1 checkstyle 1m 38s trunk passed
          +1 mvnsite 4m 2s trunk passed
          +1 mvneclipse 1m 58s trunk passed
          +1 findbugs 6m 12s trunk passed
          +1 javadoc 2m 5s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 49s the patch passed
          +1 compile 7m 29s the patch passed
          +1 cc 7m 29s the patch passed
          -1 javac 7m 29s root generated 2 new + 700 unchanged - 2 fixed = 702 total (was 702)
          -1 checkstyle 1m 45s root: The patch generated 37 new + 958 unchanged - 14 fixed = 995 total (was 972)
          +1 mvnsite 3m 36s the patch passed
          +1 mvneclipse 1m 42s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 5m 39s the patch passed
          +1 javadoc 0m 18s hadoop-yarn-api in the patch passed.
          +1 javadoc 0m 26s hadoop-yarn-common in the patch passed.
          +1 javadoc 0m 17s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 236 unchanged - 1 fixed = 236 total (was 237)
          +1 javadoc 0m 20s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 javadoc 0m 12s hadoop-yarn-server-tests in the patch passed.
          +1 javadoc 0m 15s hadoop-yarn-client in the patch passed.
          +1 javadoc 0m 12s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 unit 0m 25s hadoop-yarn-api in the patch passed.
          +1 unit 2m 17s hadoop-yarn-common in the patch passed.
          -1 unit 12m 14s hadoop-yarn-server-nodemanager in the patch failed.
          +1 unit 40m 5s hadoop-yarn-server-resourcemanager in the patch passed.
          -1 unit 4m 28s hadoop-yarn-server-tests in the patch failed.
          +1 unit 16m 11s hadoop-yarn-client in the patch passed.
          +1 unit 104m 12s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 asflicense 0m 31s The patch does not generate ASF License warnings.
          240m 44s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.containermanager.container.TestContainer
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833858/YARN-4597.003.patch
          JIRA Issue YARN-4597
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux 65917dba80c5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b61fb26
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/13417/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13417/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13417/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13417/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13417/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/13417/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13417/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13417/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 19 new or modified test files. 0 mvndep 0m 42s Maven dependency ordering for branch +1 mvninstall 7m 21s trunk passed +1 compile 9m 14s trunk passed +1 checkstyle 1m 38s trunk passed +1 mvnsite 4m 2s trunk passed +1 mvneclipse 1m 58s trunk passed +1 findbugs 6m 12s trunk passed +1 javadoc 2m 5s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 49s the patch passed +1 compile 7m 29s the patch passed +1 cc 7m 29s the patch passed -1 javac 7m 29s root generated 2 new + 700 unchanged - 2 fixed = 702 total (was 702) -1 checkstyle 1m 45s root: The patch generated 37 new + 958 unchanged - 14 fixed = 995 total (was 972) +1 mvnsite 3m 36s the patch passed +1 mvneclipse 1m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 5m 39s the patch passed +1 javadoc 0m 18s hadoop-yarn-api in the patch passed. +1 javadoc 0m 26s hadoop-yarn-common in the patch passed. +1 javadoc 0m 17s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 236 unchanged - 1 fixed = 236 total (was 237) +1 javadoc 0m 20s hadoop-yarn-server-resourcemanager in the patch passed. +1 javadoc 0m 12s hadoop-yarn-server-tests in the patch passed. +1 javadoc 0m 15s hadoop-yarn-client in the patch passed. +1 javadoc 0m 12s hadoop-mapreduce-client-jobclient in the patch passed. +1 unit 0m 25s hadoop-yarn-api in the patch passed. +1 unit 2m 17s hadoop-yarn-common in the patch passed. -1 unit 12m 14s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 40m 5s hadoop-yarn-server-resourcemanager in the patch passed. -1 unit 4m 28s hadoop-yarn-server-tests in the patch failed. +1 unit 16m 11s hadoop-yarn-client in the patch passed. +1 unit 104m 12s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 240m 44s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.container.TestContainer   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833858/YARN-4597.003.patch JIRA Issue YARN-4597 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux 65917dba80c5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b61fb26 Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/13417/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13417/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13417/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13417/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13417/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/13417/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13417/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13417/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          Arun Suresh, some more questions and comments on the patch:

          • why are these two transitions added?
                .addTransition(ContainerState.DONE, ContainerState.DONE,
                    ContainerEventType.CONTAINER_RESOURCES_CLEANEDUP)
                .addTransition(ContainerState.DONE, ContainerState.DONE,
                    ContainerEventType.CONTAINER_LAUNCHED)
                .addTransition(ContainerState.DONE, ContainerState.DONE,
            
          • the storeContainerKilled will be called in ContainerLaunch#cleanupContainer later, we don't need to call it here?
              public void sendKillEvent(int exitStatus, String description) {
                try {
                  context.getNMStateStore().storeContainerKilled(containerId);
                } catch (IOException ioe) {
                  LOG.error("Could not log container state change to state store..", ioe);
                }
            
          • remove unused imports in ContainersMonitor.java
          • remove ContainersMonitorImpl#allocatedCpuUsage unused method
          • why do you need to add the additional check for SCHEDULED state ?
                  // Process running containers
                  if (remoteContainer.getState() == ContainerState.RUNNING ||
                      remoteContainer.getState() == ContainerState.SCHEDULED) {
                      
          • why this test needs to be changed?
                    testGetContainerStatus(container, i,
                        EnumSet.of(ContainerState.RUNNING, ContainerState.SCHEDULED), "",
            
          • similarly here in TestNodeManagerShutdown, we still need to change the test to make sure the container reaches to running state?
                Assert.assertTrue(
                    EnumSet.of(ContainerState.RUNNING, ContainerState.SCHEDULED)
                        .contains(containerStatus.getState()));
            
          • why do we need to change the test to run for 10 min ?
              @Test(timeout = 600000)
              public void testAMRMClient() throws YarnException, IOException {
            
          • unreleated to this patch: should ResourceUtilization#pmem,vmem be changed to long type ? we had specifically changed it for Resource object
          • we don't need to synchronize on the currentUtilization object? I don't see any other place it's synchronized
            synchronized (currentUtilization) {
            
          • In case we exceed the max-queue length, we are killing the container directly instead of queueing the container, in this case, we should not store the container as queued?
                  try {
                    this.context.getNMStateStore().storeContainerQueued(
                        container.getContainerId());
                  } catch (IOException e) {
                    LOG.warn("Could not store container state into store..", e);
                  }
            
          • The ResourceUtilizationManager looks like only incorporated some utility methods, not sure how we will make this pluggable later..
          • I think there might be some behavior change or bug for scheduling guaranteed containers when the oppotunistic-queue is enabled.
            • Previously, when launching container, NM will not check for current vmem usage, and cpu usage. It assumes what RM allocated can be launched.
            • Now, NM will check these limits and won't launch the container if hits the limit.
            • Suppose the guarateed container hits the limit, it will be queued into queuedGuaranteedContainers. And this container will never be launched until one other container finishes which triggers the code path, even if it's not hitting these limits any more. This is a problem especially when other containers are long running container and never finishes.
          • The logic to select opportunisitic container: we may kill more opportunistic containers than required. e.g.
            • one guarateed container comes, we select one opportunistic container
            • before the selected opportunistic container is killed, another guarateed comes, then we will select two opportunistic containers to kill
            • The process repeats, we may end up killing more opportunistic containers than required.
          Show
          jianhe Jian He added a comment - Arun Suresh , some more questions and comments on the patch: why are these two transitions added? .addTransition(ContainerState.DONE, ContainerState.DONE, ContainerEventType.CONTAINER_RESOURCES_CLEANEDUP) .addTransition(ContainerState.DONE, ContainerState.DONE, ContainerEventType.CONTAINER_LAUNCHED) .addTransition(ContainerState.DONE, ContainerState.DONE, the storeContainerKilled will be called in ContainerLaunch#cleanupContainer later, we don't need to call it here? public void sendKillEvent( int exitStatus, String description) { try { context.getNMStateStore().storeContainerKilled(containerId); } catch (IOException ioe) { LOG.error( "Could not log container state change to state store.." , ioe); } remove unused imports in ContainersMonitor.java remove ContainersMonitorImpl#allocatedCpuUsage unused method why do you need to add the additional check for SCHEDULED state ? // Process running containers if (remoteContainer.getState() == ContainerState.RUNNING || remoteContainer.getState() == ContainerState.SCHEDULED) { why this test needs to be changed? testGetContainerStatus(container, i, EnumSet.of(ContainerState.RUNNING, ContainerState.SCHEDULED), "", similarly here in TestNodeManagerShutdown, we still need to change the test to make sure the container reaches to running state? Assert.assertTrue( EnumSet.of(ContainerState.RUNNING, ContainerState.SCHEDULED) .contains(containerStatus.getState())); why do we need to change the test to run for 10 min ? @Test(timeout = 600000) public void testAMRMClient() throws YarnException, IOException { unreleated to this patch: should ResourceUtilization#pmem,vmem be changed to long type ? we had specifically changed it for Resource object we don't need to synchronize on the currentUtilization object? I don't see any other place it's synchronized synchronized (currentUtilization) { In case we exceed the max-queue length, we are killing the container directly instead of queueing the container, in this case, we should not store the container as queued? try { this .context.getNMStateStore().storeContainerQueued( container.getContainerId()); } catch (IOException e) { LOG.warn( "Could not store container state into store.." , e); } The ResourceUtilizationManager looks like only incorporated some utility methods, not sure how we will make this pluggable later.. I think there might be some behavior change or bug for scheduling guaranteed containers when the oppotunistic-queue is enabled. Previously, when launching container, NM will not check for current vmem usage, and cpu usage. It assumes what RM allocated can be launched. Now, NM will check these limits and won't launch the container if hits the limit. Suppose the guarateed container hits the limit, it will be queued into queuedGuaranteedContainers. And this container will never be launched until one other container finishes which triggers the code path, even if it's not hitting these limits any more. This is a problem especially when other containers are long running container and never finishes. The logic to select opportunisitic container: we may kill more opportunistic containers than required. e.g. one guarateed container comes, we select one opportunistic container before the selected opportunistic container is killed, another guarateed comes, then we will select two opportunistic containers to kill The process repeats, we may end up killing more opportunistic containers than required.
          Hide
          asuresh Arun Suresh added a comment - - edited

          Jian He, thanks again for taking a look.

          I think there might be some behavior change or bug for scheduling guaranteed containers when the oppotunistic-queue is enabled. Previously, when launching container, NM will not check for current vmem usage, and cpu usage. It assumes what RM allocated can be launched. Now, NM will check these limits and won't launch the container if hits the limit.

          Yup, we do a hasResources check only at the start of a container and when a container is killed. We assumed that resources requested by a container is constant, essentially we considered only actual allocated resources which we assume will not varying during the lifetime of the container... which implies, there is no point in checking this at any other time other than start and kill of containers.
          But like you stated, if we consider container resource utilization, based on the work Karthik Kambatla is doing in YARN-1011, then yes, we should have a timer thread that periodically checks the vmem and cpu usage and starts (and kills) containers based on that.

          the ResourceUtilizationManager looks like only incorporated some utility methods, not sure how we will make this pluggable later.

          Following on my point above, the idea was to have a ResourceUtilizationManager that can provide a different value of getCurrentUtilization, addResource and subtractResource which is used by the ContainerScheduler to calculate the resources to free up. For instance, the current default one only takes into account actual resource allocated to containers... for YARN-1011, we might replace that with the resource utilized by running containers, and provide a different value for getCurrentUtilization. The timer thread I mentioned in the previous point, which can be apart of this new ResourceUtilizationManager, can send events to the scheduler to re-process queued containers when utilization has changed.

          The logic to select opportunisitic container: we may kill more opportunistic containers than required. e.g...

          Good catch, in the resourcesToFreeUp, I needed to decrement any already-marked-for-kill opportunistic container. It was there earlier, Had removed it when I was testing something, but forgot to put it back

          we don't need to synchronize on the currentUtilization object? I don't see any other place it's synchronized

          Yup, It isnt required. Varun did point out the same.. I thought I had fixed it, think I might have missed 'git add'ing the change

          w.r.t Adding the new transitions, I was seeing some error messages in some testcases. Will rerun and see if they are required… but in anycase, having them there should be harmless right?

          The rest of your comments makes sense.. will address them shortly.

          Show
          asuresh Arun Suresh added a comment - - edited Jian He , thanks again for taking a look. I think there might be some behavior change or bug for scheduling guaranteed containers when the oppotunistic-queue is enabled. Previously, when launching container, NM will not check for current vmem usage, and cpu usage. It assumes what RM allocated can be launched. Now, NM will check these limits and won't launch the container if hits the limit. Yup, we do a hasResources check only at the start of a container and when a container is killed. We assumed that resources requested by a container is constant, essentially we considered only actual allocated resources which we assume will not varying during the lifetime of the container... which implies, there is no point in checking this at any other time other than start and kill of containers. But like you stated, if we consider container resource utilization , based on the work Karthik Kambatla is doing in YARN-1011 , then yes, we should have a timer thread that periodically checks the vmem and cpu usage and starts (and kills) containers based on that. the ResourceUtilizationManager looks like only incorporated some utility methods, not sure how we will make this pluggable later. Following on my point above, the idea was to have a ResourceUtilizationManager that can provide a different value of getCurrentUtilization , addResource and subtractResource which is used by the ContainerScheduler to calculate the resources to free up. For instance, the current default one only takes into account actual resource allocated to containers... for YARN-1011 , we might replace that with the resource utilized by running containers, and provide a different value for getCurrentUtilization . The timer thread I mentioned in the previous point, which can be apart of this new ResourceUtilizationManager, can send events to the scheduler to re-process queued containers when utilization has changed. The logic to select opportunisitic container: we may kill more opportunistic containers than required. e.g... Good catch, in the resourcesToFreeUp , I needed to decrement any already-marked-for-kill opportunistic container. It was there earlier, Had removed it when I was testing something, but forgot to put it back we don't need to synchronize on the currentUtilization object? I don't see any other place it's synchronized Yup, It isnt required. Varun did point out the same.. I thought I had fixed it, think I might have missed 'git add'ing the change w.r.t Adding the new transitions, I was seeing some error messages in some testcases. Will rerun and see if they are required… but in anycase, having them there should be harmless right? The rest of your comments makes sense.. will address them shortly.
          Hide
          jianhe Jian He added a comment -

          the current default one only takes into account actual resource allocated to containers..We assumed that resources requested by a container is constant,

          oh, I thought it's checking for the dynamically changing vmem, it's actually only checking for the max-limit. then we are fine I think.

          Show
          jianhe Jian He added a comment - the current default one only takes into account actual resource allocated to containers..We assumed that resources requested by a container is constant, oh, I thought it's checking for the dynamically changing vmem, it's actually only checking for the max-limit. then we are fine I think.
          Hide
          jianhe Jian He added a comment -

          Will rerun and see if they are required… but in anycase, having them there should be harmless right?

          If the events should not happen in theory, I think we should not add it. otherwise, it could possibly hide bugs..we should fix the source to not send the events, instead of ignoring the events at receiver.

          Show
          jianhe Jian He added a comment - Will rerun and see if they are required… but in anycase, having them there should be harmless right? If the events should not happen in theory, I think we should not add it. otherwise, it could possibly hide bugs..we should fix the source to not send the events, instead of ignoring the events at receiver.
          Hide
          asuresh Arun Suresh added a comment - - edited

          Updated the patch based on Jian He's suggestions:

          1. Reverted the un-necessary state transistions
          2. Reverted test cases to wait for RUNNING
          3. Removed un-used imports.
          4. Fixed test- cases
          5. Added some javadoc for the ResourceUtilizationManager

          I've also updated the pull request. Attached the consolidated patch here to kick of Jenkins.

          why do you need to add the additional check for SCHEDULED state ?

                // Process running containers
                if (remoteContainer.getState() == ContainerState.RUNNING ||
                    remoteContainer.getState() == ContainerState.SCHEDULED) {
          

          This is actually needed. We need the RM to move the container out of the ACQUIRED state. Currently, it happens only if container is in the RUNNING state. We need to do so for SCHEDULED state as well.

          Show
          asuresh Arun Suresh added a comment - - edited Updated the patch based on Jian He 's suggestions: Reverted the un-necessary state transistions Reverted test cases to wait for RUNNING Removed un-used imports. Fixed test- cases Added some javadoc for the ResourceUtilizationManager I've also updated the pull request. Attached the consolidated patch here to kick of Jenkins. why do you need to add the additional check for SCHEDULED state ? // Process running containers if (remoteContainer.getState() == ContainerState.RUNNING || remoteContainer.getState() == ContainerState.SCHEDULED) { This is actually needed. We need the RM to move the container out of the ACQUIRED state. Currently, it happens only if container is in the RUNNING state. We need to do so for SCHEDULED state as well.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 19 new or modified test files.
          0 mvndep 1m 39s Maven dependency ordering for branch
          +1 mvninstall 7m 43s trunk passed
          +1 compile 8m 41s trunk passed
          +1 checkstyle 2m 2s trunk passed
          +1 mvnsite 4m 4s trunk passed
          +1 mvneclipse 2m 4s trunk passed
          +1 findbugs 5m 38s trunk passed
          +1 javadoc 2m 30s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 3m 21s the patch passed
          +1 compile 8m 19s the patch passed
          +1 cc 8m 19s the patch passed
          -1 javac 8m 19s root generated 2 new + 700 unchanged - 2 fixed = 702 total (was 702)
          -1 checkstyle 1m 54s root: The patch generated 36 new + 1017 unchanged - 15 fixed = 1053 total (was 1032)
          +1 mvnsite 3m 52s the patch passed
          +1 mvneclipse 1m 51s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 6m 42s the patch passed
          +1 javadoc 0m 21s hadoop-yarn-api in the patch passed.
          +1 javadoc 0m 32s hadoop-yarn-common in the patch passed.
          +1 javadoc 0m 20s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 236 unchanged - 1 fixed = 236 total (was 237)
          +1 javadoc 0m 30s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 javadoc 0m 13s hadoop-yarn-server-tests in the patch passed.
          +1 javadoc 0m 19s hadoop-yarn-client in the patch passed.
          +1 javadoc 0m 17s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 unit 0m 32s hadoop-yarn-api in the patch passed.
          +1 unit 2m 36s hadoop-yarn-common in the patch passed.
          +1 unit 13m 3s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 39m 34s hadoop-yarn-server-resourcemanager in the patch passed.
          -1 unit 4m 29s hadoop-yarn-server-tests in the patch failed.
          -1 unit 16m 10s hadoop-yarn-client in the patch failed.
          +1 unit 99m 57s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          242m 18s



          Reason Tests
          Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.client.cli.TestLogsCLI



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue YARN-4597
          GITHUB PR https://github.com/apache/hadoop/pull/143
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux 60b018ee3192 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d4725bf
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/13481/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13481/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13481/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13481/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13481/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt https://builds.apache.org/job/PreCommit-YARN-Build/13481/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13481/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13481/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 19 new or modified test files. 0 mvndep 1m 39s Maven dependency ordering for branch +1 mvninstall 7m 43s trunk passed +1 compile 8m 41s trunk passed +1 checkstyle 2m 2s trunk passed +1 mvnsite 4m 4s trunk passed +1 mvneclipse 2m 4s trunk passed +1 findbugs 5m 38s trunk passed +1 javadoc 2m 30s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 3m 21s the patch passed +1 compile 8m 19s the patch passed +1 cc 8m 19s the patch passed -1 javac 8m 19s root generated 2 new + 700 unchanged - 2 fixed = 702 total (was 702) -1 checkstyle 1m 54s root: The patch generated 36 new + 1017 unchanged - 15 fixed = 1053 total (was 1032) +1 mvnsite 3m 52s the patch passed +1 mvneclipse 1m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 6m 42s the patch passed +1 javadoc 0m 21s hadoop-yarn-api in the patch passed. +1 javadoc 0m 32s hadoop-yarn-common in the patch passed. +1 javadoc 0m 20s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 236 unchanged - 1 fixed = 236 total (was 237) +1 javadoc 0m 30s hadoop-yarn-server-resourcemanager in the patch passed. +1 javadoc 0m 13s hadoop-yarn-server-tests in the patch passed. +1 javadoc 0m 19s hadoop-yarn-client in the patch passed. +1 javadoc 0m 17s hadoop-mapreduce-client-jobclient in the patch passed. +1 unit 0m 32s hadoop-yarn-api in the patch passed. +1 unit 2m 36s hadoop-yarn-common in the patch passed. +1 unit 13m 3s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 39m 34s hadoop-yarn-server-resourcemanager in the patch passed. -1 unit 4m 29s hadoop-yarn-server-tests in the patch failed. -1 unit 16m 10s hadoop-yarn-client in the patch failed. +1 unit 99m 57s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 242m 18s Reason Tests Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.client.cli.TestLogsCLI Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-4597 GITHUB PR https://github.com/apache/hadoop/pull/143 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux 60b018ee3192 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d4725bf Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/13481/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13481/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13481/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13481/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13481/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt https://builds.apache.org/job/PreCommit-YARN-Build/13481/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13481/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13481/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment -

          The test failures are unrelated to the patch.
          Will fix the javadoc and checkstyle shortly.

          Please note: Container Increase/Decrease does not currently go thru the Scheduler. Think it is better working on that in a subsequent patch, once we are in agreement over this patch.

          Show
          asuresh Arun Suresh added a comment - The test failures are unrelated to the patch. Will fix the javadoc and checkstyle shortly. Please note: Container Increase/Decrease does not currently go thru the Scheduler. Think it is better working on that in a subsequent patch, once we are in agreement over this patch.
          Hide
          asuresh Arun Suresh added a comment -

          Updated patch v005 (also updated the Pull Request).

          • Added testcase to verify the situation noted by Jian He is correctly handled and does not happen:

            The logic to select opportunisitic container: we may kill more opportunistic containers than required. e.g...

          • Added some more javadocs and fixed some checkstyles.
          • Rebased with trunk.
          Show
          asuresh Arun Suresh added a comment - Updated patch v005 (also updated the Pull Request). Added testcase to verify the situation noted by Jian He is correctly handled and does not happen: The logic to select opportunisitic container: we may kill more opportunistic containers than required. e.g... Added some more javadocs and fixed some checkstyles. Rebased with trunk.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 19 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 43s trunk passed
          +1 compile 6m 55s trunk passed
          +1 checkstyle 1m 40s trunk passed
          +1 mvnsite 3m 21s trunk passed
          +1 mvneclipse 1m 46s trunk passed
          +1 findbugs 5m 9s trunk passed
          +1 javadoc 2m 3s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 49s the patch passed
          +1 compile 7m 17s the patch passed
          +1 cc 7m 17s the patch passed
          -1 javac 7m 17s root generated 2 new + 701 unchanged - 2 fixed = 703 total (was 703)
          -1 checkstyle 1m 40s root: The patch generated 7 new + 1018 unchanged - 15 fixed = 1025 total (was 1033)
          +1 mvnsite 3m 16s the patch passed
          +1 mvneclipse 1m 41s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 5m 45s the patch passed
          +1 javadoc 0m 18s hadoop-yarn-api in the patch passed.
          +1 javadoc 0m 28s hadoop-yarn-common in the patch passed.
          +1 javadoc 0m 18s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 236 unchanged - 1 fixed = 236 total (was 237)
          +1 javadoc 0m 20s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 javadoc 0m 11s hadoop-yarn-server-tests in the patch passed.
          +1 javadoc 0m 15s hadoop-yarn-client in the patch passed.
          +1 javadoc 0m 13s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed.
          +1 unit 2m 17s hadoop-yarn-common in the patch passed.
          +1 unit 12m 53s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 36m 23s hadoop-yarn-server-resourcemanager in the patch passed.
          -1 unit 4m 28s hadoop-yarn-server-tests in the patch failed.
          -1 unit 16m 12s hadoop-yarn-client in the patch failed.
          +1 unit 103m 4s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          230m 41s



          Reason Tests
          Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.client.cli.TestLogsCLI



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue YARN-4597
          GITHUB PR https://github.com/apache/hadoop/pull/143
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux 61f28e49ce91 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b18f35f
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/13486/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13486/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13486/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13486/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13486/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt https://builds.apache.org/job/PreCommit-YARN-Build/13486/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13486/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13486/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 19 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 43s trunk passed +1 compile 6m 55s trunk passed +1 checkstyle 1m 40s trunk passed +1 mvnsite 3m 21s trunk passed +1 mvneclipse 1m 46s trunk passed +1 findbugs 5m 9s trunk passed +1 javadoc 2m 3s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 49s the patch passed +1 compile 7m 17s the patch passed +1 cc 7m 17s the patch passed -1 javac 7m 17s root generated 2 new + 701 unchanged - 2 fixed = 703 total (was 703) -1 checkstyle 1m 40s root: The patch generated 7 new + 1018 unchanged - 15 fixed = 1025 total (was 1033) +1 mvnsite 3m 16s the patch passed +1 mvneclipse 1m 41s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 5m 45s the patch passed +1 javadoc 0m 18s hadoop-yarn-api in the patch passed. +1 javadoc 0m 28s hadoop-yarn-common in the patch passed. +1 javadoc 0m 18s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 236 unchanged - 1 fixed = 236 total (was 237) +1 javadoc 0m 20s hadoop-yarn-server-resourcemanager in the patch passed. +1 javadoc 0m 11s hadoop-yarn-server-tests in the patch passed. +1 javadoc 0m 15s hadoop-yarn-client in the patch passed. +1 javadoc 0m 13s hadoop-mapreduce-client-jobclient in the patch passed. +1 unit 0m 26s hadoop-yarn-api in the patch passed. +1 unit 2m 17s hadoop-yarn-common in the patch passed. +1 unit 12m 53s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 36m 23s hadoop-yarn-server-resourcemanager in the patch passed. -1 unit 4m 28s hadoop-yarn-server-tests in the patch failed. -1 unit 16m 12s hadoop-yarn-client in the patch failed. +1 unit 103m 4s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 230m 41s Reason Tests Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.client.cli.TestLogsCLI Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-4597 GITHUB PR https://github.com/apache/hadoop/pull/143 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux 61f28e49ce91 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b18f35f Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/13486/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13486/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13486/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13486/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13486/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt https://builds.apache.org/job/PreCommit-YARN-Build/13486/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13486/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13486/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          Few more comments

          • maybe rename ContainerScheduler#runningContainers to scheduledContainers
          • The ContainerLaunch#killedBeforeStart flag, looks like the exising flag 'shouldLaunchContainer' serves the same purpose, can we reuse that ? if so, the container#isMarkedToKill is also not needed.
          • NodeManager#containerScheduler variable not used, remove
          • I think this comment is not addressed ? "In case we exceed the max-queue length, we are killing the container directly instead of queueing the container, in this case, we should not store the container as queued?"
          Show
          jianhe Jian He added a comment - Few more comments maybe rename ContainerScheduler#runningContainers to scheduledContainers The ContainerLaunch#killedBeforeStart flag, looks like the exising flag 'shouldLaunchContainer' serves the same purpose, can we reuse that ? if so, the container#isMarkedToKill is also not needed. NodeManager#containerScheduler variable not used, remove I think this comment is not addressed ? "In case we exceed the max-queue length, we are killing the container directly instead of queueing the container, in this case, we should not store the container as queued?"
          Hide
          asuresh Arun Suresh added a comment - - edited

          Rebased patch after YARN-2995 and latest commits to trunk.

          Jian He, thanks for the comments..

          maybe rename ContainerScheduler#runningContainers to scheduledContainers

          Given that the SCHEDULED state is a state that comes before RUNNING. and the runningContainers collection actually holds containers that have been marked to run by the scheduler, am thinking scheduled containers may not be apt here. How about scheduledToRunContainers ?

          The ContainerLaunch#killedBeforeStart flag, looks like the exising flag 'shouldLaunchContainer' serves the same purpose, can we reuse that ? if so, the container#isMarkedToKill is also not needed.

          Hmm.. my understanding is that it is slightly different. The shouldLaunchContainer is IIUC, used during the recovery process to signal that the container should not be launched. What killedBeforeStart aims to do is to notify the ContainerLaunch (which runs in a different thread) that the Scheduler, which had requested to start the container earlier, in the last moment changed its mind and decided to kill it. Using shouldLaunchContainer also causes the CONTAINER_LAUNCH event to be fired which I do not want.

          NodeManager#containerScheduler variable not used, remove

          Done

          I think this comment is not addressed ? "In case we exceed the max-queue length, we are killing the container directly instead of queueing the container, in this case, we should not store the container as queued?"

          Yeah.. meant to comment on it. This is actually the desired behavior. Once queue limit is reached, no new opportunistic containers should also be queued. The AM is free to request it again. The MRAppMaster, for eg. re-requests the same task as a GUARANTEED container.

          Hope this made sense ?

          Show
          asuresh Arun Suresh added a comment - - edited Rebased patch after YARN-2995 and latest commits to trunk. Jian He , thanks for the comments.. maybe rename ContainerScheduler#runningContainers to scheduledContainers Given that the SCHEDULED state is a state that comes before RUNNING. and the runningContainers collection actually holds containers that have been marked to run by the scheduler, am thinking scheduled containers may not be apt here. How about scheduledToRunContainers ? The ContainerLaunch#killedBeforeStart flag, looks like the exising flag 'shouldLaunchContainer' serves the same purpose, can we reuse that ? if so, the container#isMarkedToKill is also not needed. Hmm.. my understanding is that it is slightly different. The shouldLaunchContainer is IIUC, used during the recovery process to signal that the container should not be launched. What killedBeforeStart aims to do is to notify the ContainerLaunch (which runs in a different thread) that the Scheduler, which had requested to start the container earlier, in the last moment changed its mind and decided to kill it. Using shouldLaunchContainer also causes the CONTAINER_LAUNCH event to be fired which I do not want. NodeManager#containerScheduler variable not used, remove Done I think this comment is not addressed ? "In case we exceed the max-queue length, we are killing the container directly instead of queueing the container, in this case, we should not store the container as queued?" Yeah.. meant to comment on it. This is actually the desired behavior. Once queue limit is reached, no new opportunistic containers should also be queued. The AM is free to request it again. The MRAppMaster, for eg. re-requests the same task as a GUARANTEED container. Hope this made sense ?
          Hide
          jianhe Jian He added a comment -

          scheduledToRunContainers

          sounds good

          Using shouldLaunchContainer also causes the CONTAINER_LAUNCH event to be fired which I do not want.

          I think the shouldLaunchContainer is also used for in case container killed has been called, skip launching the container. (see the code comment in below code: "// Check if the container is signalled to be killed." ). In terms of the additional event, I also think that should be changed for existing code. The code for sending container_launched event should be inside the 'else' block when really launching the container.

              // LaunchContainer is a blocking call. We are here almost means the
              // container is launched, so send out the event.
              dispatcher.getEventHandler().handle(new ContainerEvent(
                  containerId,
                  ContainerEventType.CONTAINER_LAUNCHED));
              context.getNMStateStore().storeContainerLaunched(containerId);
          
              // Check if the container is signalled to be killed.
              if (!shouldLaunchContainer.compareAndSet(false, true)) {
                LOG.info("Container " + containerId + " not launched as "
                    + "cleanup already called");
                return ExitCode.TERMINATED.getExitCode();
          

          Once queue limit is reached, no new opportunistic containers should also be queued. The AM is free to request it again. The MRAppMaster, for eg. re-requests the same task as a GUARANTEED container.

          sorry for unclear, I meant below code. It will call 'storeContainerQueued' unconditionally, even though the container is not queued when it reached the queue-len limit. shouldn't we not call storeContainerQueued in that case ?

          {
                try {
                  this.context.getNMStateStore().storeContainerQueued(
                      container.getContainerId());
                } catch (IOException e) {
                  LOG.warn("Could not store container state into store..", e);
                }
                LOG.info("No available resources for container {} to start its execution "
                    + "immediately.", container.getContainerId());
                if (container.getContainerTokenIdentifier().getExecutionType() ==
                    ExecutionType.GUARANTEED) {
                  queuedGuaranteedContainers.put(container.getContainerId(), container);
                  // Kill running opportunistic containers to make space for
                  // guaranteed container.
                  killOpportunisticContainers(container);
                } else {
                  if (queuedOpportunisticContainers.size() <= maxOppQueueLength) {
                    LOG.info("Opportunistic container {} will be queued at the NM.",
                        container.getContainerId());
                    queuedOpportunisticContainers.put(
                        container.getContainerId(), container);
                  } else {
                    LOG.info("Opportunistic container [{}] will not be queued at the NM" +
                        "since max queue length [{}] has been reached",
                        container.getContainerId(), maxOppQueueLength);
                    container.sendKillEvent(
                        ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
                        "Opportunistic container queue is full.");
                  }
                }
              }
          
          Show
          jianhe Jian He added a comment - scheduledToRunContainers sounds good Using shouldLaunchContainer also causes the CONTAINER_LAUNCH event to be fired which I do not want. I think the shouldLaunchContainer is also used for in case container killed has been called, skip launching the container. (see the code comment in below code: "// Check if the container is signalled to be killed." ). In terms of the additional event, I also think that should be changed for existing code. The code for sending container_launched event should be inside the 'else' block when really launching the container. // LaunchContainer is a blocking call. We are here almost means the // container is launched, so send out the event. dispatcher.getEventHandler().handle( new ContainerEvent( containerId, ContainerEventType.CONTAINER_LAUNCHED)); context.getNMStateStore().storeContainerLaunched(containerId); // Check if the container is signalled to be killed. if (!shouldLaunchContainer.compareAndSet( false , true )) { LOG.info( "Container " + containerId + " not launched as " + "cleanup already called" ); return ExitCode.TERMINATED.getExitCode(); Once queue limit is reached, no new opportunistic containers should also be queued. The AM is free to request it again. The MRAppMaster, for eg. re-requests the same task as a GUARANTEED container. sorry for unclear, I meant below code. It will call 'storeContainerQueued' unconditionally, even though the container is not queued when it reached the queue-len limit. shouldn't we not call storeContainerQueued in that case ? { try { this .context.getNMStateStore().storeContainerQueued( container.getContainerId()); } catch (IOException e) { LOG.warn( "Could not store container state into store.." , e); } LOG.info( "No available resources for container {} to start its execution " + "immediately." , container.getContainerId()); if (container.getContainerTokenIdentifier().getExecutionType() == ExecutionType.GUARANTEED) { queuedGuaranteedContainers.put(container.getContainerId(), container); // Kill running opportunistic containers to make space for // guaranteed container. killOpportunisticContainers(container); } else { if (queuedOpportunisticContainers.size() <= maxOppQueueLength) { LOG.info( "Opportunistic container {} will be queued at the NM." , container.getContainerId()); queuedOpportunisticContainers.put( container.getContainerId(), container); } else { LOG.info( "Opportunistic container [{}] will not be queued at the NM" + "since max queue length [{}] has been reached" , container.getContainerId(), maxOppQueueLength); container.sendKillEvent( ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, "Opportunistic container queue is full." ); } } }
          Hide
          asuresh Arun Suresh added a comment -

          I also think that should be changed for existing code. The code for sending container_launched event should be inside the 'else' block when really launching the container.

          Aah.. I was worried I would break something else If I moved it inside the else. If it is fine to modify the existing path, I'll use the 'shouldLaunchContainer' flag.. Thanks.

          It will call 'storeContainerQueued' unconditionally, even though the container is not queued when it reached the queue-len limit. shouldn't we not call storeContainerQueued in that case.

          Got it.. Yup, will fix it.

          Show
          asuresh Arun Suresh added a comment - I also think that should be changed for existing code. The code for sending container_launched event should be inside the 'else' block when really launching the container. Aah.. I was worried I would break something else If I moved it inside the else. If it is fine to modify the existing path, I'll use the 'shouldLaunchContainer' flag.. Thanks. It will call 'storeContainerQueued' unconditionally, even though the container is not queued when it reached the queue-len limit. shouldn't we not call storeContainerQueued in that case. Got it.. Yup, will fix it.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 19 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 44s trunk passed
          +1 compile 11m 4s trunk passed
          +1 checkstyle 1m 52s trunk passed
          +1 mvnsite 4m 42s trunk passed
          +1 mvneclipse 2m 49s trunk passed
          +1 findbugs 6m 55s trunk passed
          +1 javadoc 3m 30s trunk passed
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 3m 3s the patch passed
          +1 compile 9m 35s the patch passed
          +1 cc 9m 35s the patch passed
          +1 javac 9m 35s the patch passed
          -0 checkstyle 1m 56s root: The patch generated 11 new + 1050 unchanged - 16 fixed = 1061 total (was 1066)
          +1 mvnsite 4m 54s the patch passed
          +1 mvneclipse 3m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 7m 38s the patch passed
          +1 javadoc 0m 33s hadoop-yarn-api in the patch passed.
          +1 javadoc 0m 46s hadoop-yarn-common in the patch passed.
          +1 javadoc 0m 31s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236)
          +1 javadoc 0m 39s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 javadoc 0m 25s hadoop-yarn-server-tests in the patch passed.
          +1 javadoc 0m 30s hadoop-yarn-client in the patch passed.
          +1 javadoc 0m 27s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 unit 0m 42s hadoop-yarn-api in the patch passed.
          +1 unit 2m 44s hadoop-yarn-common in the patch passed.
          +1 unit 13m 6s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 75m 1s hadoop-yarn-server-resourcemanager in the patch failed.
          -1 unit 4m 47s hadoop-yarn-server-tests in the patch failed.
          +1 unit 16m 41s hadoop-yarn-client in the patch passed.
          -1 unit 383m 32s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 1m 40s The patch does not generate ASF License warnings.
          596m 45s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
            hadoop.yarn.server.resourcemanager.TestSignalContainer
            hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
            hadoop.yarn.server.resourcemanager.TestRMNodeTransitions
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
            hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
            hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler
            hadoop.yarn.server.resourcemanager.TestApplicationCleanup
            hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
            hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher
            hadoop.yarn.server.resourcemanager.TestRM
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
            hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
            hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures
            hadoop.yarn.server.resourcemanager.TestResourceTrackerService
            hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.mapreduce.v2.TestMRJobsWithProfiler
            hadoop.mapred.TestMRTimelineEventHandling
            hadoop.mapreduce.v2.TestMiniMRProxyUser
            hadoop.mapreduce.v2.TestMRJobsWithHistoryService
            hadoop.mapred.TestClusterMRNotification
          Timed out junit tests org.apache.hadoop.mapreduce.TestMRJobClient
            org.apache.hadoop.mapreduce.v2.TestUberAM
            org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
            org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
            org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
            org.apache.hadoop.mapred.TestJobCounters
            org.apache.hadoop.mapreduce.v2.TestMRJobs
            org.apache.hadoop.mapreduce.TestLargeSort
            org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
            org.apache.hadoop.mapred.TestJobCleanup
            org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
            org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
            org.apache.hadoop.mapred.TestMiniMRClientCluster
            org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
            org.apache.hadoop.mapred.TestLazyOutput
            org.apache.hadoop.mapred.TestReduceFetch
            org.apache.hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities
            org.apache.hadoop.mapred.TestMerge



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue YARN-4597
          GITHUB PR https://github.com/apache/hadoop/pull/143
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux 8e2dd736b2d5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / de3b4aa
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13812/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13812/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13812/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13812/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13812/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13812/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 19 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 44s trunk passed +1 compile 11m 4s trunk passed +1 checkstyle 1m 52s trunk passed +1 mvnsite 4m 42s trunk passed +1 mvneclipse 2m 49s trunk passed +1 findbugs 6m 55s trunk passed +1 javadoc 3m 30s trunk passed 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 3m 3s the patch passed +1 compile 9m 35s the patch passed +1 cc 9m 35s the patch passed +1 javac 9m 35s the patch passed -0 checkstyle 1m 56s root: The patch generated 11 new + 1050 unchanged - 16 fixed = 1061 total (was 1066) +1 mvnsite 4m 54s the patch passed +1 mvneclipse 3m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 7m 38s the patch passed +1 javadoc 0m 33s hadoop-yarn-api in the patch passed. +1 javadoc 0m 46s hadoop-yarn-common in the patch passed. +1 javadoc 0m 31s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236) +1 javadoc 0m 39s hadoop-yarn-server-resourcemanager in the patch passed. +1 javadoc 0m 25s hadoop-yarn-server-tests in the patch passed. +1 javadoc 0m 30s hadoop-yarn-client in the patch passed. +1 javadoc 0m 27s hadoop-mapreduce-client-jobclient in the patch passed. +1 unit 0m 42s hadoop-yarn-api in the patch passed. +1 unit 2m 44s hadoop-yarn-common in the patch passed. +1 unit 13m 6s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 75m 1s hadoop-yarn-server-resourcemanager in the patch failed. -1 unit 4m 47s hadoop-yarn-server-tests in the patch failed. +1 unit 16m 41s hadoop-yarn-client in the patch passed. -1 unit 383m 32s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 1m 40s The patch does not generate ASF License warnings. 596m 45s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestSignalContainer   hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler   hadoop.yarn.server.resourcemanager.TestRMNodeTransitions   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter   hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler   hadoop.yarn.server.resourcemanager.TestApplicationCleanup   hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart   hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher   hadoop.yarn.server.resourcemanager.TestRM   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps   hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage   hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures   hadoop.yarn.server.resourcemanager.TestResourceTrackerService   hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.mapreduce.v2.TestMRJobsWithProfiler   hadoop.mapred.TestMRTimelineEventHandling   hadoop.mapreduce.v2.TestMiniMRProxyUser   hadoop.mapreduce.v2.TestMRJobsWithHistoryService   hadoop.mapred.TestClusterMRNotification Timed out junit tests org.apache.hadoop.mapreduce.TestMRJobClient   org.apache.hadoop.mapreduce.v2.TestUberAM   org.apache.hadoop.mapred.TestMRIntermediateDataEncryption   org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution   org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner   org.apache.hadoop.mapred.TestJobCounters   org.apache.hadoop.mapreduce.v2.TestMRJobs   org.apache.hadoop.mapreduce.TestLargeSort   org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter   org.apache.hadoop.mapred.TestJobCleanup   org.apache.hadoop.mapreduce.TestMapReduceLazyOutput   org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers   org.apache.hadoop.mapred.TestMiniMRClientCluster   org.apache.hadoop.mapred.TestReduceFetchFromPartialMem   org.apache.hadoop.mapred.TestLazyOutput   org.apache.hadoop.mapred.TestReduceFetch   org.apache.hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities   org.apache.hadoop.mapred.TestMerge Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue YARN-4597 GITHUB PR https://github.com/apache/hadoop/pull/143 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux 8e2dd736b2d5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / de3b4aa Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13812/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13812/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13812/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13812/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13812/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13812/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment -

          Updating patch to fix some rebasing errors.

          Jian He, I tried to reuse shouldLaunchContainer for the killedBeforeStart flag, but I see too many test failures, since a lot of the code expects the CONTAINER_LAUNCH event to also be raised. If its ok with you, I will give it another look once we converge on everything else.

          I've handled your remaining comment.

          Show
          asuresh Arun Suresh added a comment - Updating patch to fix some rebasing errors. Jian He , I tried to reuse shouldLaunchContainer for the killedBeforeStart flag, but I see too many test failures, since a lot of the code expects the CONTAINER_LAUNCH event to also be raised. If its ok with you, I will give it another look once we converge on everything else. I've handled your remaining comment.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 3m 47s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 19 new or modified test files.
          0 mvndep 1m 50s Maven dependency ordering for branch
          +1 mvninstall 7m 5s trunk passed
          +1 compile 10m 46s trunk passed
          +1 checkstyle 1m 49s trunk passed
          +1 mvnsite 4m 23s trunk passed
          +1 mvneclipse 2m 52s trunk passed
          +1 findbugs 6m 8s trunk passed
          +1 javadoc 3m 22s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 2m 55s the patch passed
          +1 compile 9m 17s the patch passed
          +1 cc 9m 17s the patch passed
          +1 javac 9m 17s the patch passed
          -0 checkstyle 1m 55s root: The patch generated 12 new + 1049 unchanged - 14 fixed = 1061 total (was 1063)
          +1 mvnsite 4m 48s the patch passed
          +1 mvneclipse 3m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 7m 21s the patch passed
          +1 javadoc 0m 32s hadoop-yarn-api in the patch passed.
          +1 javadoc 0m 42s hadoop-yarn-common in the patch passed.
          +1 javadoc 0m 32s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236)
          +1 javadoc 0m 35s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 javadoc 0m 26s hadoop-yarn-server-tests in the patch passed.
          +1 javadoc 0m 29s hadoop-yarn-client in the patch passed.
          +1 javadoc 0m 28s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 unit 0m 40s hadoop-yarn-api in the patch passed.
          +1 unit 2m 31s hadoop-yarn-common in the patch passed.
          +1 unit 12m 41s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 35m 55s hadoop-yarn-server-resourcemanager in the patch passed.
          -1 unit 4m 53s hadoop-yarn-server-tests in the patch failed.
          +1 unit 16m 13s hadoop-yarn-client in the patch passed.
          +1 unit 96m 26s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 asflicense 0m 59s The patch does not generate ASF License warnings.
          271m 22s



          Reason Tests
          Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.server.TestContainerManagerSecurity



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue YARN-4597
          GITHUB PR https://github.com/apache/hadoop/pull/143
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux 19d2186f530b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f38a6d0
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13822/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13822/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13822/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13822/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 3m 47s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 19 new or modified test files. 0 mvndep 1m 50s Maven dependency ordering for branch +1 mvninstall 7m 5s trunk passed +1 compile 10m 46s trunk passed +1 checkstyle 1m 49s trunk passed +1 mvnsite 4m 23s trunk passed +1 mvneclipse 2m 52s trunk passed +1 findbugs 6m 8s trunk passed +1 javadoc 3m 22s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 2m 55s the patch passed +1 compile 9m 17s the patch passed +1 cc 9m 17s the patch passed +1 javac 9m 17s the patch passed -0 checkstyle 1m 55s root: The patch generated 12 new + 1049 unchanged - 14 fixed = 1061 total (was 1063) +1 mvnsite 4m 48s the patch passed +1 mvneclipse 3m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 7m 21s the patch passed +1 javadoc 0m 32s hadoop-yarn-api in the patch passed. +1 javadoc 0m 42s hadoop-yarn-common in the patch passed. +1 javadoc 0m 32s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236) +1 javadoc 0m 35s hadoop-yarn-server-resourcemanager in the patch passed. +1 javadoc 0m 26s hadoop-yarn-server-tests in the patch passed. +1 javadoc 0m 29s hadoop-yarn-client in the patch passed. +1 javadoc 0m 28s hadoop-mapreduce-client-jobclient in the patch passed. +1 unit 0m 40s hadoop-yarn-api in the patch passed. +1 unit 2m 31s hadoop-yarn-common in the patch passed. +1 unit 12m 41s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 35m 55s hadoop-yarn-server-resourcemanager in the patch passed. -1 unit 4m 53s hadoop-yarn-server-tests in the patch failed. +1 unit 16m 13s hadoop-yarn-client in the patch passed. +1 unit 96m 26s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 59s The patch does not generate ASF License warnings. 271m 22s Reason Tests Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestContainerManagerSecurity Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue YARN-4597 GITHUB PR https://github.com/apache/hadoop/pull/143 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux 19d2186f530b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f38a6d0 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13822/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13822/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13822/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13822/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          I tried to reuse shouldLaunchContainer for the killedBeforeStart flag, but I see too many test failures, since a lot of the code expects the CONTAINER_LAUNCH event to also be raised. If its ok with you, I will give it another look once we converge on everything else.

          Sure, other things look good to me.

          Show
          jianhe Jian He added a comment - I tried to reuse shouldLaunchContainer for the killedBeforeStart flag, but I see too many test failures, since a lot of the code expects the CONTAINER_LAUNCH event to also be raised. If its ok with you, I will give it another look once we converge on everything else. Sure, other things look good to me.
          Hide
          asuresh Arun Suresh added a comment -

          Jian He, tried another round at refactoring out the 'killedBeforeStart' flag from the ContainerLaunch. I now feel, it should not be clubbed with the 'shouldLaunchContainer' flag. They signify very different states: the former is used to ensure the container is never started while the latter is used to capture the event that the container has already been launch, and thereby 'should not be launched again'. I feel a better name for 'shouldLaunchContainer' should have been 'containerAlreadyLaunched'.

          I have raised YARN-5860 and YARN-5861 to track the remaining tasks need to get this truly feature complete. If Jian He is fine with the current state of this patch, I would like to check this in. It would be nice though if Konstantinos Karanasos also takes a look at this to verify if I havn't missed anything in the ContainerScheduler functionality vis-a-vis the QueuingContainerManagerImpl

          Show
          asuresh Arun Suresh added a comment - Jian He , tried another round at refactoring out the 'killedBeforeStart' flag from the ContainerLaunch. I now feel, it should not be clubbed with the 'shouldLaunchContainer' flag. They signify very different states: the former is used to ensure the container is never started while the latter is used to capture the event that the container has already been launch, and thereby 'should not be launched again'. I feel a better name for 'shouldLaunchContainer' should have been 'containerAlreadyLaunched'. I have raised YARN-5860 and YARN-5861 to track the remaining tasks need to get this truly feature complete. If Jian He is fine with the current state of this patch, I would like to check this in. It would be nice though if Konstantinos Karanasos also takes a look at this to verify if I havn't missed anything in the ContainerScheduler functionality vis-a-vis the QueuingContainerManagerImpl
          Hide
          jianhe Jian He added a comment -

          I see.. thanks for looking into it.. I guess the killedBeforeStart flag could be avoided, by calling "container.isMarkedToKill()" directly in the handleContainerExitCode method like below ?

                if (!container.isMarkedToKill()) {
                  dispatcher.getEventHandler().handle(
                      new ContainerExitEvent(containerId,
                          ContainerEventType.CONTAINER_KILLED_ON_REQUEST, exitCode,
                          diagnosticInfo.toString()));
                }
          
          Show
          jianhe Jian He added a comment - I see.. thanks for looking into it.. I guess the killedBeforeStart flag could be avoided, by calling "container.isMarkedToKill()" directly in the handleContainerExitCode method like below ? if (!container.isMarkedToKill()) { dispatcher.getEventHandler().handle( new ContainerExitEvent(containerId, ContainerEventType.CONTAINER_KILLED_ON_REQUEST, exitCode, diagnosticInfo.toString())); }
          Hide
          asuresh Arun Suresh added a comment -

          Hmmm.. I agree it looks equivalent. But what if the 'isMarkedForKill' flag is set on the container object by the scheduler in between the time the ContainerLaunch thread is running the 'ContainerLaunch::launchContainer()' method and the 'ContainerLaunch::handleContainerExitCode()'. In that case, the CONTAINER_KILLED_ON_REQUEST event will not be sent.

          Show
          asuresh Arun Suresh added a comment - Hmmm.. I agree it looks equivalent. But what if the 'isMarkedForKill' flag is set on the container object by the scheduler in between the time the ContainerLaunch thread is running the 'ContainerLaunch::launchContainer()' method and the 'ContainerLaunch::handleContainerExitCode()'. In that case, the CONTAINER_KILLED_ON_REQUEST event will not be sent.
          Hide
          jianhe Jian He added a comment -

          I see, sounds reasonable.

          I feel a better name for 'shouldLaunchContainer' should have been 'containerAlreadyLaunched'.

          btw. agree with this, in case you would like to change it.

          Show
          jianhe Jian He added a comment - I see, sounds reasonable. I feel a better name for 'shouldLaunchContainer' should have been 'containerAlreadyLaunched'. btw. agree with this, in case you would like to change it.
          Hide
          asuresh Arun Suresh added a comment -

          Updated patch.

          btw. agree with this, in case you would like to change it.

          Done..

          Show
          asuresh Arun Suresh added a comment - Updated patch. btw. agree with this, in case you would like to change it. Done..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 8s YARN-4597 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue YARN-4597
          GITHUB PR https://github.com/apache/hadoop/pull/143
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13834/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 8s YARN-4597 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-4597 GITHUB PR https://github.com/apache/hadoop/pull/143 Console output https://builds.apache.org/job/PreCommit-YARN-Build/13834/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment -

          Rebasing with trunk again..

          Show
          asuresh Arun Suresh added a comment - Rebasing with trunk again..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 6s YARN-4597 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue YARN-4597
          GITHUB PR https://github.com/apache/hadoop/pull/143
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13839/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s YARN-4597 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-4597 GITHUB PR https://github.com/apache/hadoop/pull/143 Console output https://builds.apache.org/job/PreCommit-YARN-Build/13839/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 9s YARN-4597 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue YARN-4597
          GITHUB PR https://github.com/apache/hadoop/pull/143
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13841/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 9s YARN-4597 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-4597 GITHUB PR https://github.com/apache/hadoop/pull/143 Console output https://builds.apache.org/job/PreCommit-YARN-Build/13841/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 19 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 7m 54s trunk passed
          +1 compile 15m 13s trunk passed
          +1 checkstyle 2m 12s trunk passed
          +1 mvnsite 5m 10s trunk passed
          +1 mvneclipse 3m 29s trunk passed
          -1 findbugs 0m 29s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml)
          +1 javadoc 3m 56s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 3m 37s the patch passed
          +1 compile 13m 4s the patch passed
          +1 cc 13m 4s the patch passed
          +1 javac 13m 4s the patch passed
          -0 checkstyle 2m 25s root: The patch generated 15 new + 1052 unchanged - 16 fixed = 1067 total (was 1068)
          +1 mvnsite 6m 16s the patch passed
          +1 mvneclipse 3m 50s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          -1 findbugs 0m 29s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml)
          +1 javadoc 0m 33s hadoop-yarn-api in the patch passed.
          +1 javadoc 0m 42s hadoop-yarn-common in the patch passed.
          +1 javadoc 0m 32s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236)
          +1 javadoc 0m 37s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 javadoc 0m 28s hadoop-yarn-server-tests in the patch passed.
          +1 javadoc 0m 27s hadoop-yarn-client in the patch passed.
          +1 javadoc 0m 30s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 unit 0m 44s hadoop-yarn-api in the patch passed.
          +1 unit 2m 38s hadoop-yarn-common in the patch passed.
          +1 unit 13m 47s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 40m 31s hadoop-yarn-server-resourcemanager in the patch passed.
          -1 unit 4m 46s hadoop-yarn-server-tests in the patch failed.
          -1 unit 31m 15s hadoop-yarn-client in the patch failed.
          -1 unit 105m 53s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 57s The patch does not generate ASF License warnings.
          312m 51s



          Reason Tests
          Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.client.api.impl.TestAMRMProxy
            hadoop.mapred.TestMROpportunisticMaps
            hadoop.mapred.pipes.TestPipeApplication
          Timed out junit tests org.apache.hadoop.yarn.client.api.impl.TestDistributedScheduling



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue YARN-4597
          GITHUB PR https://github.com/apache/hadoop/pull/143
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux 1182f904bf2b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 283fa33
          Default Java 1.8.0_101
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13842/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13842/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13842/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13842/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13842/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13842/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13842/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13842/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 19 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 7m 54s trunk passed +1 compile 15m 13s trunk passed +1 checkstyle 2m 12s trunk passed +1 mvnsite 5m 10s trunk passed +1 mvneclipse 3m 29s trunk passed -1 findbugs 0m 29s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml) +1 javadoc 3m 56s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 3m 37s the patch passed +1 compile 13m 4s the patch passed +1 cc 13m 4s the patch passed +1 javac 13m 4s the patch passed -0 checkstyle 2m 25s root: The patch generated 15 new + 1052 unchanged - 16 fixed = 1067 total (was 1068) +1 mvnsite 6m 16s the patch passed +1 mvneclipse 3m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 0m 29s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml) +1 javadoc 0m 33s hadoop-yarn-api in the patch passed. +1 javadoc 0m 42s hadoop-yarn-common in the patch passed. +1 javadoc 0m 32s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236) +1 javadoc 0m 37s hadoop-yarn-server-resourcemanager in the patch passed. +1 javadoc 0m 28s hadoop-yarn-server-tests in the patch passed. +1 javadoc 0m 27s hadoop-yarn-client in the patch passed. +1 javadoc 0m 30s hadoop-mapreduce-client-jobclient in the patch passed. +1 unit 0m 44s hadoop-yarn-api in the patch passed. +1 unit 2m 38s hadoop-yarn-common in the patch passed. +1 unit 13m 47s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 40m 31s hadoop-yarn-server-resourcemanager in the patch passed. -1 unit 4m 46s hadoop-yarn-server-tests in the patch failed. -1 unit 31m 15s hadoop-yarn-client in the patch failed. -1 unit 105m 53s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 57s The patch does not generate ASF License warnings. 312m 51s Reason Tests Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.client.api.impl.TestAMRMProxy   hadoop.mapred.TestMROpportunisticMaps   hadoop.mapred.pipes.TestPipeApplication Timed out junit tests org.apache.hadoop.yarn.client.api.impl.TestDistributedScheduling Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue YARN-4597 GITHUB PR https://github.com/apache/hadoop/pull/143 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux 1182f904bf2b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 283fa33 Default Java 1.8.0_101 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13842/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13842/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13842/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13842/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13842/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13842/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13842/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13842/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment - - edited

          The new test failures are caused by an issue with YARN-5833. Kicking Jenkins again after resolving it..

          Show
          asuresh Arun Suresh added a comment - - edited The new test failures are caused by an issue with YARN-5833 . Kicking Jenkins again after resolving it..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 19 new or modified test files.
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 7m 37s trunk passed
          +1 compile 12m 25s trunk passed
          +1 checkstyle 1m 52s trunk passed
          +1 mvnsite 4m 46s trunk passed
          +1 mvneclipse 2m 55s trunk passed
          -1 findbugs 0m 29s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml)
          +1 javadoc 3m 24s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 3m 0s the patch passed
          +1 compile 10m 2s the patch passed
          +1 cc 10m 2s the patch passed
          +1 javac 10m 2s the patch passed
          -0 checkstyle 1m 58s root: The patch generated 15 new + 1053 unchanged - 16 fixed = 1068 total (was 1069)
          +1 mvnsite 5m 5s the patch passed
          +1 mvneclipse 3m 17s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          -1 findbugs 0m 30s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml)
          +1 javadoc 0m 32s hadoop-yarn-api in the patch passed.
          +1 javadoc 0m 41s hadoop-yarn-common in the patch passed.
          +1 javadoc 0m 32s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236)
          +1 javadoc 0m 36s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 javadoc 0m 26s hadoop-yarn-server-tests in the patch passed.
          +1 javadoc 0m 30s hadoop-yarn-client in the patch passed.
          +1 javadoc 0m 28s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 unit 0m 41s hadoop-yarn-api in the patch passed.
          +1 unit 2m 34s hadoop-yarn-common in the patch passed.
          +1 unit 12m 54s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 36m 28s hadoop-yarn-server-resourcemanager in the patch passed.
          -1 unit 4m 39s hadoop-yarn-server-tests in the patch failed.
          +1 unit 16m 22s hadoop-yarn-client in the patch passed.
          -1 unit 102m 27s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 58s The patch does not generate ASF License warnings.
          277m 14s



          Reason Tests
          Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.mapred.pipes.TestPipeApplication



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue YARN-4597
          GITHUB PR https://github.com/apache/hadoop/pull/143
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux d16684e78adc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 280357c
          Default Java 1.8.0_101
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13845/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13845/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13845/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13845/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13845/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13845/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13845/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 19 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 7m 37s trunk passed +1 compile 12m 25s trunk passed +1 checkstyle 1m 52s trunk passed +1 mvnsite 4m 46s trunk passed +1 mvneclipse 2m 55s trunk passed -1 findbugs 0m 29s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml) +1 javadoc 3m 24s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 3m 0s the patch passed +1 compile 10m 2s the patch passed +1 cc 10m 2s the patch passed +1 javac 10m 2s the patch passed -0 checkstyle 1m 58s root: The patch generated 15 new + 1053 unchanged - 16 fixed = 1068 total (was 1069) +1 mvnsite 5m 5s the patch passed +1 mvneclipse 3m 17s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 0m 30s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml) +1 javadoc 0m 32s hadoop-yarn-api in the patch passed. +1 javadoc 0m 41s hadoop-yarn-common in the patch passed. +1 javadoc 0m 32s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236) +1 javadoc 0m 36s hadoop-yarn-server-resourcemanager in the patch passed. +1 javadoc 0m 26s hadoop-yarn-server-tests in the patch passed. +1 javadoc 0m 30s hadoop-yarn-client in the patch passed. +1 javadoc 0m 28s hadoop-mapreduce-client-jobclient in the patch passed. +1 unit 0m 41s hadoop-yarn-api in the patch passed. +1 unit 2m 34s hadoop-yarn-common in the patch passed. +1 unit 12m 54s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 36m 28s hadoop-yarn-server-resourcemanager in the patch passed. -1 unit 4m 39s hadoop-yarn-server-tests in the patch failed. +1 unit 16m 22s hadoop-yarn-client in the patch passed. -1 unit 102m 27s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 58s The patch does not generate ASF License warnings. 277m 14s Reason Tests Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.mapred.pipes.TestPipeApplication Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue YARN-4597 GITHUB PR https://github.com/apache/hadoop/pull/143 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux d16684e78adc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 280357c Default Java 1.8.0_101 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13845/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13845/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13845/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13845/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13845/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13845/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13845/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          the patch looks good to me, is Konstantinos Karanasos going to take a look ?

          Show
          jianhe Jian He added a comment - the patch looks good to me, is Konstantinos Karanasos going to take a look ?
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Hi Jian He. Yes, I will check the patch today.

          Show
          kkaranasos Konstantinos Karanasos added a comment - Hi Jian He . Yes, I will check the patch today.
          Hide
          kasha Karthik Kambatla added a comment -

          Arun Suresh - could you hold off until Friday evening? I ll try and get to it by then. Thanks.

          Show
          kasha Karthik Kambatla added a comment - Arun Suresh - could you hold off until Friday evening? I ll try and get to it by then. Thanks.
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Thanks for working on this, Arun Suresh! I am sending some first comments. I have not yet looked at the ContainerScheduler – I will do that tomorrow.

          • The Container has two new methods (sendLaunchEvent and sendKillEvent), which are public and are not following the design of the rest of the code that keeps such methods private and calls them through transitions in the ContainerImpl. Let's try to use the existing design if possible.
          • In RMNodeImpl:
            • Instead of using the launchedContainers for both the launched and the queued, we might want to split it in two: one for the launched and one for the queued containers.
            • I think we should not add opportunistic containers to the launchContainers. If we do, they will be added to the newlyLaunchedContainers, then to the nodeUpdateQueue, and, if I am not wrong, they will be propagated to the schedulers for the guaranteed containers, which will create problems. I have to look at it a bit more, but my hunch is that we should avoid doing it. Even if it does not affect the resource accounting, I don't see any advantage to adding them.
          • In the OpportunisticContainerAllocatorAMService we are now calling the SchedulerNode::allocate, and then we do not update the used resources, but we do update some other counters, which leads to inconsistencies. For example, when releasing a container, I think at the moment we are not calling the release of the SchedulerNode, which means that the container count will become inconsistent.
            • Instead, I suggest to add some counters for opportunistic containers at the SchedulerNode, both for the number of containers and for the resources used. In this case, we need to make sure that those resources are released too.
          • Maybe as part of a different JIRA, we should at some point extend the container.metrics in the ContainerImpl to keep track of the scheduled/queued containers.
          Nits:
          • There seem to be two redundant parameters at YarnConfiguration at the moment: NM_CONTAINER_QUEUING_MIN_QUEUE_LENGTH and NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH. If I am not missing something, we should keep one of the two.
          • yarn-default.xml: numbed -> number (in a comment)
          • TestNodeManagerResync: I think it is better to use one of the existing methods for waiting to get to the RUNNING state.
          • In Container/ContainerImpl and all the associated classes, I would suggest to rename isMarkedToKill to isMarkedForKilling. I know it is minor, but it is more self-explanatory.

          I will send more comments once I check the ContainerScheduler.
          Also, let's stress-test the code in a cluster before committing to make sure everything is good. I can help with that.

          Show
          kkaranasos Konstantinos Karanasos added a comment - Thanks for working on this, Arun Suresh ! I am sending some first comments. I have not yet looked at the ContainerScheduler – I will do that tomorrow. The Container has two new methods ( sendLaunchEvent and sendKillEvent ), which are public and are not following the design of the rest of the code that keeps such methods private and calls them through transitions in the ContainerImpl . Let's try to use the existing design if possible. In RMNodeImpl : Instead of using the launchedContainers for both the launched and the queued, we might want to split it in two: one for the launched and one for the queued containers. I think we should not add opportunistic containers to the launchContainers . If we do, they will be added to the newlyLaunchedContainers , then to the nodeUpdateQueue , and, if I am not wrong, they will be propagated to the schedulers for the guaranteed containers, which will create problems. I have to look at it a bit more, but my hunch is that we should avoid doing it. Even if it does not affect the resource accounting, I don't see any advantage to adding them. In the OpportunisticContainerAllocatorAMService we are now calling the SchedulerNode::allocate , and then we do not update the used resources, but we do update some other counters, which leads to inconsistencies. For example, when releasing a container, I think at the moment we are not calling the release of the SchedulerNode , which means that the container count will become inconsistent. Instead, I suggest to add some counters for opportunistic containers at the SchedulerNode , both for the number of containers and for the resources used. In this case, we need to make sure that those resources are released too. Maybe as part of a different JIRA, we should at some point extend the container.metrics in the ContainerImpl to keep track of the scheduled/queued containers. Nits: There seem to be two redundant parameters at YarnConfiguration at the moment: NM_CONTAINER_QUEUING_MIN_QUEUE_LENGTH and NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH . If I am not missing something, we should keep one of the two. yarn-default.xml : numbed -> number (in a comment) TestNodeManagerResync : I think it is better to use one of the existing methods for waiting to get to the RUNNING state. In Container / ContainerImpl and all the associated classes, I would suggest to rename isMarkedToKill to isMarkedForKilling . I know it is minor, but it is more self-explanatory. I will send more comments once I check the ContainerScheduler . Also, let's stress-test the code in a cluster before committing to make sure everything is good. I can help with that.
          Hide
          asuresh Arun Suresh added a comment - - edited

          Appreciate the review Konstantinos Karanasos,

          1.

          The Container has two new methods (sendLaunchEvent and sendKillEvent), which are public and are not following..

          sendKillEvent is used by the Scheduler (which is in another package) to kill a container. Since this patch introduces an external entity that launches and kills a container, viz. the Scheduler, I feel it is apt to keep both as public methods. I prefer it to 'dispatcher.getEventHandler().handle..'.

          2.
          The Container needs to be added to the nodeUpdateQueue if the container is to be move from ACQUIRED to RUNNING state (this is a state transition all containers should go thru). Regarding the launchedContainers, Lets have both Opportunistic and Guaranteed containers flow through a common code-path... and introduce specific behaviors if required in subsequent patches as and when required.

          3.

          In the OpportunisticContainerAllocatorAMService we are now calling the SchedulerNode::allocate, and then we do not update the used resources but we do update some other counters, which leads to inconsistencies.

          Hmmm... I do see that the numContainers are not decremented correctly when release. Thanks... but it looks like it would more likely just impact reporting / UI, nothing functional (Will update the patch). Can you specify which other counters specifically ? Like I mentioned above.. lets run all containers thru as much of the common code path before we add new counters etc.

          4.

          Maybe as part of a different JIRA, we should at some point extend the container.metrics in the ContainerImpl to keep track of the scheduled/queued containers.

          Yup.. +1 to that.

          The rest of your comments make sense... will update patch.

          let's stress-test the code in a cluster before committing to make sure everything is good

          It has been tested on a 3 node cluster and MR Pi jobs (with opportunistic containers) and I didn't hit any major issues. We can always open follow-up JIRAs for specific performance related issues as and when we find it. Besides, stess-testing is not really a precondition to committing a patch.

          Show
          asuresh Arun Suresh added a comment - - edited Appreciate the review Konstantinos Karanasos , 1. The Container has two new methods (sendLaunchEvent and sendKillEvent), which are public and are not following.. sendKillEvent is used by the Scheduler (which is in another package) to kill a container. Since this patch introduces an external entity that launches and kills a container, viz. the Scheduler, I feel it is apt to keep both as public methods. I prefer it to 'dispatcher.getEventHandler().handle..'. 2. The Container needs to be added to the nodeUpdateQueue if the container is to be move from ACQUIRED to RUNNING state (this is a state transition all containers should go thru). Regarding the launchedContainers , Lets have both Opportunistic and Guaranteed containers flow through a common code-path... and introduce specific behaviors if required in subsequent patches as and when required. 3. In the OpportunisticContainerAllocatorAMService we are now calling the SchedulerNode::allocate, and then we do not update the used resources but we do update some other counters, which leads to inconsistencies. Hmmm... I do see that the numContainers are not decremented correctly when release. Thanks... but it looks like it would more likely just impact reporting / UI, nothing functional (Will update the patch). Can you specify which other counters specifically ? Like I mentioned above.. lets run all containers thru as much of the common code path before we add new counters etc. 4. Maybe as part of a different JIRA, we should at some point extend the container.metrics in the ContainerImpl to keep track of the scheduled/queued containers. Yup.. +1 to that. The rest of your comments make sense... will update patch. let's stress-test the code in a cluster before committing to make sure everything is good It has been tested on a 3 node cluster and MR Pi jobs (with opportunistic containers) and I didn't hit any major issues. We can always open follow-up JIRAs for specific performance related issues as and when we find it. Besides, stess-testing is not really a precondition to committing a patch.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87638283

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          + new HashMap<>();
          +
          + // Containers launched by the Scheduler will take a while to actually
          + // move to the RUNNING state, but should still be fair game for killing
          + // by the scheduler to make room for guaranteed containers.
          + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers =
          — End diff –

          Are these running containers? If yes, should we call them so?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87638283 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedOpportunisticContainers = new LinkedHashMap<>(); + + // Used to keep track of containers that have been marked to be killed + // to make room for a guaranteed container. + private final Map<ContainerId, Container> oppContainersMarkedForKill = + new HashMap<>(); + + // Containers launched by the Scheduler will take a while to actually + // move to the RUNNING state, but should still be fair game for killing + // by the scheduler to make room for guaranteed containers. + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers = — End diff – Are these running containers? If yes, should we call them so?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87628936

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          — End diff –

          s/it/its

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87628936 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are — End diff – s/it/its
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87639403

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          + new HashMap<>();
          +
          + // Containers launched by the Scheduler will take a while to actually
          + // move to the RUNNING state, but should still be fair game for killing
          + // by the scheduler to make room for guaranteed containers.
          + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers =
          + new LinkedHashMap<>();
          +
          + private final ContainerQueuingLimit queuingLimit =
          + ContainerQueuingLimit.newInstance();
          +
          + private final OpportunisticContainersStatus opportunisticContainersStatus;
          +
          + // Resource Utilization Manager that decides how utilization of the cluster
          + // increase / decreases based on container start / finish
          + private ResourceUtilizationManager utilizationManager;
          +
          + /**
          + * Instantiate a Container Scheduler.
          + * @param context NodeManager Context.
          + */
          + public ContainerScheduler(Context context)

          { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + }

          +
          + @VisibleForTesting
          + public ContainerScheduler(Context context, int qLength)

          { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + }

          +
          + /**
          + * Handle ContainerSchedulerEvents.
          + * @param event ContainerSchedulerEvent.
          + */
          + @Override
          + public void handle(ContainerSchedulerEvent event) {
          + switch (event.getType())

          { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + }

          + }
          +
          + /**
          + * Return number of queued containers.
          + * @return Number of queued containers.
          + */
          + public int getNumQueuedContainers()

          { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedGuaranteedContainers()

          { + return this.queuedGuaranteedContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedOpportunisticContainers()

          { + return this.queuedOpportunisticContainers.size(); + }

          +
          + public OpportunisticContainersStatus getOpportunisticContainersStatus()

          { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + }

          +
          + private void onContainerCompleted(Container container) {
          + // decrement only if it was a running container
          + if (scheduledToRunContainers.containsKey(container.getContainerId())) {
          + this.utilizationManager.subtractContainerResource(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + }

          + }
          + scheduledToRunContainers.remove(container.getContainerId());
          + oppContainersMarkedForKill.remove(container.getContainerId());
          + startPendingContainers();
          + }
          +
          + private void startPendingContainers() {
          + // Start pending guaranteed containers, if resources available.
          + boolean resourcesAvailable =
          + startContainersFromQueue(queuedGuaranteedContainers.values());
          + // Start opportunistic containers, if resources available.
          + if (resourcesAvailable)

          { + startContainersFromQueue(queuedOpportunisticContainers.values()); + }

          + }
          +
          + private boolean startContainersFromQueue(
          + Collection<Container> queuedContainers) {
          + Iterator<Container> cIter = queuedContainers.iterator();
          + boolean resourcesAvailable = true;
          + while (cIter.hasNext() && resourcesAvailable) {
          + Container container = cIter.next();
          + if (this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + cIter.remove(); + }

          else

          { + resourcesAvailable = false; + }

          + }
          + return resourcesAvailable;
          + }
          +
          + @VisibleForTesting
          + protected void scheduleContainer(Container container) {
          + if (maxOppQueueLength <= 0)

          { + startAllocatedContainer(container); + return; + }

          + if (queuedGuaranteedContainers.isEmpty() &&
          + queuedOpportunisticContainers.isEmpty() &&
          + this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + }

          else {
          + LOG.info("No available resources for container {} to start its execution "
          + + "immediately.", container.getContainerId());
          + boolean isQueued = true;
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.GUARANTEED)

          { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + }

          else {
          + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) {
          + LOG.info("Opportunistic container {} will be queued at the NM.",
          + container.getContainerId());
          + queuedOpportunisticContainers.put(
          + container.getContainerId(), container);
          + } else {
          + isQueued = false;
          + LOG.info("Opportunistic container [{}] will not be queued at the NM" +
          + "since max queue length [{}] has been reached",
          + container.getContainerId(), maxOppQueueLength);
          + container.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Opportunistic container queue is full.");
          + }
          + }
          + if (isQueued) {
          + try

          { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + }

          catch (IOException e)

          { + LOG.warn("Could not store container state into store..", e); + }

          + }
          + }
          + }
          +
          + private void killOpportunisticContainers(Container container) {
          + List<Container> extraOpportContainersToKill =
          + pickOpportunisticContainersToKill(container.getContainerId());
          + // Kill the opportunistic containers that were chosen.
          + for (Container contToKill : extraOpportContainersToKill) {
          + contToKill.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Container Killed to make room for Guaranteed Container.");
          + oppContainersMarkedForKill.put(contToKill.getContainerId(), contToKill);
          + LOG.info(
          + "Opportunistic container {} will be killed in order to start the "
          + + "execution of guaranteed container {}.",
          + contToKill.getContainerId(), container.getContainerId());
          + }
          + }
          +
          + private void startAllocatedContainer(Container container) {
          + LOG.info("Starting container [" + container.getContainerId()+ "]");
          + scheduledToRunContainers.put(container.getContainerId(), container);
          — End diff –

          If the reason we are tracking running containers is to consider them for preemption, why do we need to track guaranteed containers?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87639403 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedOpportunisticContainers = new LinkedHashMap<>(); + + // Used to keep track of containers that have been marked to be killed + // to make room for a guaranteed container. + private final Map<ContainerId, Container> oppContainersMarkedForKill = + new HashMap<>(); + + // Containers launched by the Scheduler will take a while to actually + // move to the RUNNING state, but should still be fair game for killing + // by the scheduler to make room for guaranteed containers. + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers = + new LinkedHashMap<>(); + + private final ContainerQueuingLimit queuingLimit = + ContainerQueuingLimit.newInstance(); + + private final OpportunisticContainersStatus opportunisticContainersStatus; + + // Resource Utilization Manager that decides how utilization of the cluster + // increase / decreases based on container start / finish + private ResourceUtilizationManager utilizationManager; + + /** + * Instantiate a Container Scheduler. + * @param context NodeManager Context. + */ + public ContainerScheduler(Context context) { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + } + + @VisibleForTesting + public ContainerScheduler(Context context, int qLength) { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + } + + /** + * Handle ContainerSchedulerEvents. + * @param event ContainerSchedulerEvent. + */ + @Override + public void handle(ContainerSchedulerEvent event) { + switch (event.getType()) { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + } + } + + /** + * Return number of queued containers. + * @return Number of queued containers. + */ + public int getNumQueuedContainers() { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedGuaranteedContainers() { + return this.queuedGuaranteedContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedOpportunisticContainers() { + return this.queuedOpportunisticContainers.size(); + } + + public OpportunisticContainersStatus getOpportunisticContainersStatus() { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + } + + private void onContainerCompleted(Container container) { + // decrement only if it was a running container + if (scheduledToRunContainers.containsKey(container.getContainerId())) { + this.utilizationManager.subtractContainerResource(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + } + } + scheduledToRunContainers.remove(container.getContainerId()); + oppContainersMarkedForKill.remove(container.getContainerId()); + startPendingContainers(); + } + + private void startPendingContainers() { + // Start pending guaranteed containers, if resources available. + boolean resourcesAvailable = + startContainersFromQueue(queuedGuaranteedContainers.values()); + // Start opportunistic containers, if resources available. + if (resourcesAvailable) { + startContainersFromQueue(queuedOpportunisticContainers.values()); + } + } + + private boolean startContainersFromQueue( + Collection<Container> queuedContainers) { + Iterator<Container> cIter = queuedContainers.iterator(); + boolean resourcesAvailable = true; + while (cIter.hasNext() && resourcesAvailable) { + Container container = cIter.next(); + if (this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + cIter.remove(); + } else { + resourcesAvailable = false; + } + } + return resourcesAvailable; + } + + @VisibleForTesting + protected void scheduleContainer(Container container) { + if (maxOppQueueLength <= 0) { + startAllocatedContainer(container); + return; + } + if (queuedGuaranteedContainers.isEmpty() && + queuedOpportunisticContainers.isEmpty() && + this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + } else { + LOG.info("No available resources for container {} to start its execution " + + "immediately.", container.getContainerId()); + boolean isQueued = true; + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.GUARANTEED) { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + } else { + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) { + LOG.info("Opportunistic container {} will be queued at the NM.", + container.getContainerId()); + queuedOpportunisticContainers.put( + container.getContainerId(), container); + } else { + isQueued = false; + LOG.info("Opportunistic container [{}] will not be queued at the NM" + + "since max queue length [{}] has been reached", + container.getContainerId(), maxOppQueueLength); + container.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Opportunistic container queue is full."); + } + } + if (isQueued) { + try { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + } catch (IOException e) { + LOG.warn("Could not store container state into store..", e); + } + } + } + } + + private void killOpportunisticContainers(Container container) { + List<Container> extraOpportContainersToKill = + pickOpportunisticContainersToKill(container.getContainerId()); + // Kill the opportunistic containers that were chosen. + for (Container contToKill : extraOpportContainersToKill) { + contToKill.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Container Killed to make room for Guaranteed Container."); + oppContainersMarkedForKill.put(contToKill.getContainerId(), contToKill); + LOG.info( + "Opportunistic container {} will be killed in order to start the " + + "execution of guaranteed container {}.", + contToKill.getContainerId(), container.getContainerId()); + } + } + + private void startAllocatedContainer(Container container) { + LOG.info("Starting container [" + container.getContainerId()+ "] "); + scheduledToRunContainers.put(container.getContainerId(), container); — End diff – If the reason we are tracking running containers is to consider them for preemption, why do we need to track guaranteed containers?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87639061

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          + new HashMap<>();
          +
          + // Containers launched by the Scheduler will take a while to actually
          + // move to the RUNNING state, but should still be fair game for killing
          + // by the scheduler to make room for guaranteed containers.
          + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers =
          + new LinkedHashMap<>();
          +
          + private final ContainerQueuingLimit queuingLimit =
          + ContainerQueuingLimit.newInstance();
          +
          + private final OpportunisticContainersStatus opportunisticContainersStatus;
          +
          + // Resource Utilization Manager that decides how utilization of the cluster
          + // increase / decreases based on container start / finish
          + private ResourceUtilizationManager utilizationManager;
          +
          + /**
          + * Instantiate a Container Scheduler.
          + * @param context NodeManager Context.
          + */
          + public ContainerScheduler(Context context)

          { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + }

          +
          + @VisibleForTesting
          + public ContainerScheduler(Context context, int qLength)

          { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + }

          +
          + /**
          + * Handle ContainerSchedulerEvents.
          + * @param event ContainerSchedulerEvent.
          + */
          + @Override
          + public void handle(ContainerSchedulerEvent event) {
          + switch (event.getType())

          { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + }

          + }
          +
          + /**
          + * Return number of queued containers.
          + * @return Number of queued containers.
          + */
          + public int getNumQueuedContainers()

          { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedGuaranteedContainers()

          { + return this.queuedGuaranteedContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedOpportunisticContainers()

          { + return this.queuedOpportunisticContainers.size(); + }

          +
          + public OpportunisticContainersStatus getOpportunisticContainersStatus()

          { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + }

          +
          + private void onContainerCompleted(Container container) {
          + // decrement only if it was a running container
          + if (scheduledToRunContainers.containsKey(container.getContainerId())) {
          + this.utilizationManager.subtractContainerResource(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + }

          + }
          + scheduledToRunContainers.remove(container.getContainerId());
          + oppContainersMarkedForKill.remove(container.getContainerId());
          + startPendingContainers();
          + }
          +
          + private void startPendingContainers() {
          + // Start pending guaranteed containers, if resources available.
          + boolean resourcesAvailable =
          + startContainersFromQueue(queuedGuaranteedContainers.values());
          + // Start opportunistic containers, if resources available.
          + if (resourcesAvailable)

          { + startContainersFromQueue(queuedOpportunisticContainers.values()); + }

          + }
          +
          + private boolean startContainersFromQueue(
          + Collection<Container> queuedContainers) {
          + Iterator<Container> cIter = queuedContainers.iterator();
          + boolean resourcesAvailable = true;
          + while (cIter.hasNext() && resourcesAvailable) {
          + Container container = cIter.next();
          + if (this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + cIter.remove(); + }

          else

          { + resourcesAvailable = false; + }

          + }
          + return resourcesAvailable;
          + }
          +
          + @VisibleForTesting
          + protected void scheduleContainer(Container container) {
          + if (maxOppQueueLength <= 0)

          { + startAllocatedContainer(container); + return; + }

          + if (queuedGuaranteedContainers.isEmpty() &&
          + queuedOpportunisticContainers.isEmpty() &&
          + this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + }

          else {
          + LOG.info("No available resources for container {} to start its execution "
          + + "immediately.", container.getContainerId());
          + boolean isQueued = true;
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.GUARANTEED)

          { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + }

          else {
          + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) {
          + LOG.info("Opportunistic container {} will be queued at the NM.",
          — End diff –

          Should this be at DEBUG level?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87639061 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedOpportunisticContainers = new LinkedHashMap<>(); + + // Used to keep track of containers that have been marked to be killed + // to make room for a guaranteed container. + private final Map<ContainerId, Container> oppContainersMarkedForKill = + new HashMap<>(); + + // Containers launched by the Scheduler will take a while to actually + // move to the RUNNING state, but should still be fair game for killing + // by the scheduler to make room for guaranteed containers. + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers = + new LinkedHashMap<>(); + + private final ContainerQueuingLimit queuingLimit = + ContainerQueuingLimit.newInstance(); + + private final OpportunisticContainersStatus opportunisticContainersStatus; + + // Resource Utilization Manager that decides how utilization of the cluster + // increase / decreases based on container start / finish + private ResourceUtilizationManager utilizationManager; + + /** + * Instantiate a Container Scheduler. + * @param context NodeManager Context. + */ + public ContainerScheduler(Context context) { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + } + + @VisibleForTesting + public ContainerScheduler(Context context, int qLength) { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + } + + /** + * Handle ContainerSchedulerEvents. + * @param event ContainerSchedulerEvent. + */ + @Override + public void handle(ContainerSchedulerEvent event) { + switch (event.getType()) { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + } + } + + /** + * Return number of queued containers. + * @return Number of queued containers. + */ + public int getNumQueuedContainers() { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedGuaranteedContainers() { + return this.queuedGuaranteedContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedOpportunisticContainers() { + return this.queuedOpportunisticContainers.size(); + } + + public OpportunisticContainersStatus getOpportunisticContainersStatus() { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + } + + private void onContainerCompleted(Container container) { + // decrement only if it was a running container + if (scheduledToRunContainers.containsKey(container.getContainerId())) { + this.utilizationManager.subtractContainerResource(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + } + } + scheduledToRunContainers.remove(container.getContainerId()); + oppContainersMarkedForKill.remove(container.getContainerId()); + startPendingContainers(); + } + + private void startPendingContainers() { + // Start pending guaranteed containers, if resources available. + boolean resourcesAvailable = + startContainersFromQueue(queuedGuaranteedContainers.values()); + // Start opportunistic containers, if resources available. + if (resourcesAvailable) { + startContainersFromQueue(queuedOpportunisticContainers.values()); + } + } + + private boolean startContainersFromQueue( + Collection<Container> queuedContainers) { + Iterator<Container> cIter = queuedContainers.iterator(); + boolean resourcesAvailable = true; + while (cIter.hasNext() && resourcesAvailable) { + Container container = cIter.next(); + if (this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + cIter.remove(); + } else { + resourcesAvailable = false; + } + } + return resourcesAvailable; + } + + @VisibleForTesting + protected void scheduleContainer(Container container) { + if (maxOppQueueLength <= 0) { + startAllocatedContainer(container); + return; + } + if (queuedGuaranteedContainers.isEmpty() && + queuedOpportunisticContainers.isEmpty() && + this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + } else { + LOG.info("No available resources for container {} to start its execution " + + "immediately.", container.getContainerId()); + boolean isQueued = true; + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.GUARANTEED) { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + } else { + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) { + LOG.info("Opportunistic container {} will be queued at the NM.", — End diff – Should this be at DEBUG level?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87629189

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          — End diff –

          Since only opportunistic containers will be killed to make room, can we go with a simpler name containersToKill?

          May be, oppContainersToKill?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87629189 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedOpportunisticContainers = new LinkedHashMap<>(); + + // Used to keep track of containers that have been marked to be killed + // to make room for a guaranteed container. + private final Map<ContainerId, Container> oppContainersMarkedForKill = — End diff – Since only opportunistic containers will be killed to make room, can we go with a simpler name containersToKill? May be, oppContainersToKill?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87639089

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          + new HashMap<>();
          +
          + // Containers launched by the Scheduler will take a while to actually
          + // move to the RUNNING state, but should still be fair game for killing
          + // by the scheduler to make room for guaranteed containers.
          + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers =
          + new LinkedHashMap<>();
          +
          + private final ContainerQueuingLimit queuingLimit =
          + ContainerQueuingLimit.newInstance();
          +
          + private final OpportunisticContainersStatus opportunisticContainersStatus;
          +
          + // Resource Utilization Manager that decides how utilization of the cluster
          + // increase / decreases based on container start / finish
          + private ResourceUtilizationManager utilizationManager;
          +
          + /**
          + * Instantiate a Container Scheduler.
          + * @param context NodeManager Context.
          + */
          + public ContainerScheduler(Context context)

          { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + }

          +
          + @VisibleForTesting
          + public ContainerScheduler(Context context, int qLength)

          { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + }

          +
          + /**
          + * Handle ContainerSchedulerEvents.
          + * @param event ContainerSchedulerEvent.
          + */
          + @Override
          + public void handle(ContainerSchedulerEvent event) {
          + switch (event.getType())

          { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + }

          + }
          +
          + /**
          + * Return number of queued containers.
          + * @return Number of queued containers.
          + */
          + public int getNumQueuedContainers()

          { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedGuaranteedContainers()

          { + return this.queuedGuaranteedContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedOpportunisticContainers()

          { + return this.queuedOpportunisticContainers.size(); + }

          +
          + public OpportunisticContainersStatus getOpportunisticContainersStatus()

          { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + }

          +
          + private void onContainerCompleted(Container container) {
          + // decrement only if it was a running container
          + if (scheduledToRunContainers.containsKey(container.getContainerId())) {
          + this.utilizationManager.subtractContainerResource(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + }

          + }
          + scheduledToRunContainers.remove(container.getContainerId());
          + oppContainersMarkedForKill.remove(container.getContainerId());
          + startPendingContainers();
          + }
          +
          + private void startPendingContainers() {
          + // Start pending guaranteed containers, if resources available.
          + boolean resourcesAvailable =
          + startContainersFromQueue(queuedGuaranteedContainers.values());
          + // Start opportunistic containers, if resources available.
          + if (resourcesAvailable)

          { + startContainersFromQueue(queuedOpportunisticContainers.values()); + }

          + }
          +
          + private boolean startContainersFromQueue(
          + Collection<Container> queuedContainers) {
          + Iterator<Container> cIter = queuedContainers.iterator();
          + boolean resourcesAvailable = true;
          + while (cIter.hasNext() && resourcesAvailable) {
          + Container container = cIter.next();
          + if (this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + cIter.remove(); + }

          else

          { + resourcesAvailable = false; + }

          + }
          + return resourcesAvailable;
          + }
          +
          + @VisibleForTesting
          + protected void scheduleContainer(Container container) {
          + if (maxOppQueueLength <= 0)

          { + startAllocatedContainer(container); + return; + }

          + if (queuedGuaranteedContainers.isEmpty() &&
          + queuedOpportunisticContainers.isEmpty() &&
          + this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + }

          else {
          + LOG.info("No available resources for container {} to start its execution "
          + + "immediately.", container.getContainerId());
          + boolean isQueued = true;
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.GUARANTEED)

          { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + }

          else {
          + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) {
          + LOG.info("Opportunistic container {} will be queued at the NM.",
          + container.getContainerId());
          + queuedOpportunisticContainers.put(
          + container.getContainerId(), container);
          + } else {
          + isQueued = false;
          + LOG.info("Opportunistic container [{}] will not be queued at the NM" +
          — End diff –

          Should this be at DEBUG level?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87639089 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedOpportunisticContainers = new LinkedHashMap<>(); + + // Used to keep track of containers that have been marked to be killed + // to make room for a guaranteed container. + private final Map<ContainerId, Container> oppContainersMarkedForKill = + new HashMap<>(); + + // Containers launched by the Scheduler will take a while to actually + // move to the RUNNING state, but should still be fair game for killing + // by the scheduler to make room for guaranteed containers. + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers = + new LinkedHashMap<>(); + + private final ContainerQueuingLimit queuingLimit = + ContainerQueuingLimit.newInstance(); + + private final OpportunisticContainersStatus opportunisticContainersStatus; + + // Resource Utilization Manager that decides how utilization of the cluster + // increase / decreases based on container start / finish + private ResourceUtilizationManager utilizationManager; + + /** + * Instantiate a Container Scheduler. + * @param context NodeManager Context. + */ + public ContainerScheduler(Context context) { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + } + + @VisibleForTesting + public ContainerScheduler(Context context, int qLength) { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + } + + /** + * Handle ContainerSchedulerEvents. + * @param event ContainerSchedulerEvent. + */ + @Override + public void handle(ContainerSchedulerEvent event) { + switch (event.getType()) { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + } + } + + /** + * Return number of queued containers. + * @return Number of queued containers. + */ + public int getNumQueuedContainers() { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedGuaranteedContainers() { + return this.queuedGuaranteedContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedOpportunisticContainers() { + return this.queuedOpportunisticContainers.size(); + } + + public OpportunisticContainersStatus getOpportunisticContainersStatus() { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + } + + private void onContainerCompleted(Container container) { + // decrement only if it was a running container + if (scheduledToRunContainers.containsKey(container.getContainerId())) { + this.utilizationManager.subtractContainerResource(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + } + } + scheduledToRunContainers.remove(container.getContainerId()); + oppContainersMarkedForKill.remove(container.getContainerId()); + startPendingContainers(); + } + + private void startPendingContainers() { + // Start pending guaranteed containers, if resources available. + boolean resourcesAvailable = + startContainersFromQueue(queuedGuaranteedContainers.values()); + // Start opportunistic containers, if resources available. + if (resourcesAvailable) { + startContainersFromQueue(queuedOpportunisticContainers.values()); + } + } + + private boolean startContainersFromQueue( + Collection<Container> queuedContainers) { + Iterator<Container> cIter = queuedContainers.iterator(); + boolean resourcesAvailable = true; + while (cIter.hasNext() && resourcesAvailable) { + Container container = cIter.next(); + if (this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + cIter.remove(); + } else { + resourcesAvailable = false; + } + } + return resourcesAvailable; + } + + @VisibleForTesting + protected void scheduleContainer(Container container) { + if (maxOppQueueLength <= 0) { + startAllocatedContainer(container); + return; + } + if (queuedGuaranteedContainers.isEmpty() && + queuedOpportunisticContainers.isEmpty() && + this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + } else { + LOG.info("No available resources for container {} to start its execution " + + "immediately.", container.getContainerId()); + boolean isQueued = true; + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.GUARANTEED) { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + } else { + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) { + LOG.info("Opportunistic container {} will be queued at the NM.", + container.getContainerId()); + queuedOpportunisticContainers.put( + container.getContainerId(), container); + } else { + isQueued = false; + LOG.info("Opportunistic container [{}] will not be queued at the NM" + — End diff – Should this be at DEBUG level?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87639161

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          + new HashMap<>();
          +
          + // Containers launched by the Scheduler will take a while to actually
          + // move to the RUNNING state, but should still be fair game for killing
          + // by the scheduler to make room for guaranteed containers.
          + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers =
          + new LinkedHashMap<>();
          +
          + private final ContainerQueuingLimit queuingLimit =
          + ContainerQueuingLimit.newInstance();
          +
          + private final OpportunisticContainersStatus opportunisticContainersStatus;
          +
          + // Resource Utilization Manager that decides how utilization of the cluster
          + // increase / decreases based on container start / finish
          + private ResourceUtilizationManager utilizationManager;
          +
          + /**
          + * Instantiate a Container Scheduler.
          + * @param context NodeManager Context.
          + */
          + public ContainerScheduler(Context context)

          { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + }

          +
          + @VisibleForTesting
          + public ContainerScheduler(Context context, int qLength)

          { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + }

          +
          + /**
          + * Handle ContainerSchedulerEvents.
          + * @param event ContainerSchedulerEvent.
          + */
          + @Override
          + public void handle(ContainerSchedulerEvent event) {
          + switch (event.getType())

          { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + }

          + }
          +
          + /**
          + * Return number of queued containers.
          + * @return Number of queued containers.
          + */
          + public int getNumQueuedContainers()

          { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedGuaranteedContainers()

          { + return this.queuedGuaranteedContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedOpportunisticContainers()

          { + return this.queuedOpportunisticContainers.size(); + }

          +
          + public OpportunisticContainersStatus getOpportunisticContainersStatus()

          { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + }

          +
          + private void onContainerCompleted(Container container) {
          + // decrement only if it was a running container
          + if (scheduledToRunContainers.containsKey(container.getContainerId())) {
          + this.utilizationManager.subtractContainerResource(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + }

          + }
          + scheduledToRunContainers.remove(container.getContainerId());
          + oppContainersMarkedForKill.remove(container.getContainerId());
          + startPendingContainers();
          + }
          +
          + private void startPendingContainers() {
          + // Start pending guaranteed containers, if resources available.
          + boolean resourcesAvailable =
          + startContainersFromQueue(queuedGuaranteedContainers.values());
          + // Start opportunistic containers, if resources available.
          + if (resourcesAvailable)

          { + startContainersFromQueue(queuedOpportunisticContainers.values()); + }

          + }
          +
          + private boolean startContainersFromQueue(
          + Collection<Container> queuedContainers) {
          + Iterator<Container> cIter = queuedContainers.iterator();
          + boolean resourcesAvailable = true;
          + while (cIter.hasNext() && resourcesAvailable) {
          + Container container = cIter.next();
          + if (this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + cIter.remove(); + }

          else

          { + resourcesAvailable = false; + }

          + }
          + return resourcesAvailable;
          + }
          +
          + @VisibleForTesting
          + protected void scheduleContainer(Container container) {
          + if (maxOppQueueLength <= 0)

          { + startAllocatedContainer(container); + return; + }

          + if (queuedGuaranteedContainers.isEmpty() &&
          + queuedOpportunisticContainers.isEmpty() &&
          + this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + }

          else {
          + LOG.info("No available resources for container {} to start its execution "
          + + "immediately.", container.getContainerId());
          + boolean isQueued = true;
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.GUARANTEED)

          { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + }

          else {
          + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) {
          + LOG.info("Opportunistic container {} will be queued at the NM.",
          + container.getContainerId());
          + queuedOpportunisticContainers.put(
          + container.getContainerId(), container);
          + } else {
          + isQueued = false;
          + LOG.info("Opportunistic container [{}] will not be queued at the NM" +
          + "since max queue length [{}] has been reached",
          + container.getContainerId(), maxOppQueueLength);
          + container.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Opportunistic container queue is full.");
          + }
          + }
          + if (isQueued) {
          + try

          { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + }

          catch (IOException e) {
          + LOG.warn("Could not store container state into store..", e);
          — End diff –

          Should we call out this container is queued?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87639161 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedOpportunisticContainers = new LinkedHashMap<>(); + + // Used to keep track of containers that have been marked to be killed + // to make room for a guaranteed container. + private final Map<ContainerId, Container> oppContainersMarkedForKill = + new HashMap<>(); + + // Containers launched by the Scheduler will take a while to actually + // move to the RUNNING state, but should still be fair game for killing + // by the scheduler to make room for guaranteed containers. + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers = + new LinkedHashMap<>(); + + private final ContainerQueuingLimit queuingLimit = + ContainerQueuingLimit.newInstance(); + + private final OpportunisticContainersStatus opportunisticContainersStatus; + + // Resource Utilization Manager that decides how utilization of the cluster + // increase / decreases based on container start / finish + private ResourceUtilizationManager utilizationManager; + + /** + * Instantiate a Container Scheduler. + * @param context NodeManager Context. + */ + public ContainerScheduler(Context context) { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + } + + @VisibleForTesting + public ContainerScheduler(Context context, int qLength) { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + } + + /** + * Handle ContainerSchedulerEvents. + * @param event ContainerSchedulerEvent. + */ + @Override + public void handle(ContainerSchedulerEvent event) { + switch (event.getType()) { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + } + } + + /** + * Return number of queued containers. + * @return Number of queued containers. + */ + public int getNumQueuedContainers() { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedGuaranteedContainers() { + return this.queuedGuaranteedContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedOpportunisticContainers() { + return this.queuedOpportunisticContainers.size(); + } + + public OpportunisticContainersStatus getOpportunisticContainersStatus() { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + } + + private void onContainerCompleted(Container container) { + // decrement only if it was a running container + if (scheduledToRunContainers.containsKey(container.getContainerId())) { + this.utilizationManager.subtractContainerResource(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + } + } + scheduledToRunContainers.remove(container.getContainerId()); + oppContainersMarkedForKill.remove(container.getContainerId()); + startPendingContainers(); + } + + private void startPendingContainers() { + // Start pending guaranteed containers, if resources available. + boolean resourcesAvailable = + startContainersFromQueue(queuedGuaranteedContainers.values()); + // Start opportunistic containers, if resources available. + if (resourcesAvailable) { + startContainersFromQueue(queuedOpportunisticContainers.values()); + } + } + + private boolean startContainersFromQueue( + Collection<Container> queuedContainers) { + Iterator<Container> cIter = queuedContainers.iterator(); + boolean resourcesAvailable = true; + while (cIter.hasNext() && resourcesAvailable) { + Container container = cIter.next(); + if (this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + cIter.remove(); + } else { + resourcesAvailable = false; + } + } + return resourcesAvailable; + } + + @VisibleForTesting + protected void scheduleContainer(Container container) { + if (maxOppQueueLength <= 0) { + startAllocatedContainer(container); + return; + } + if (queuedGuaranteedContainers.isEmpty() && + queuedOpportunisticContainers.isEmpty() && + this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + } else { + LOG.info("No available resources for container {} to start its execution " + + "immediately.", container.getContainerId()); + boolean isQueued = true; + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.GUARANTEED) { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + } else { + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) { + LOG.info("Opportunistic container {} will be queued at the NM.", + container.getContainerId()); + queuedOpportunisticContainers.put( + container.getContainerId(), container); + } else { + isQueued = false; + LOG.info("Opportunistic container [{}] will not be queued at the NM" + + "since max queue length [{}] has been reached", + container.getContainerId(), maxOppQueueLength); + container.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Opportunistic container queue is full."); + } + } + if (isQueued) { + try { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + } catch (IOException e) { + LOG.warn("Could not store container state into store..", e); — End diff – Should we call out this container is queued?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87639036

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          + new HashMap<>();
          +
          + // Containers launched by the Scheduler will take a while to actually
          + // move to the RUNNING state, but should still be fair game for killing
          + // by the scheduler to make room for guaranteed containers.
          + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers =
          + new LinkedHashMap<>();
          +
          + private final ContainerQueuingLimit queuingLimit =
          + ContainerQueuingLimit.newInstance();
          +
          + private final OpportunisticContainersStatus opportunisticContainersStatus;
          +
          + // Resource Utilization Manager that decides how utilization of the cluster
          + // increase / decreases based on container start / finish
          + private ResourceUtilizationManager utilizationManager;
          +
          + /**
          + * Instantiate a Container Scheduler.
          + * @param context NodeManager Context.
          + */
          + public ContainerScheduler(Context context)

          { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + }

          +
          + @VisibleForTesting
          + public ContainerScheduler(Context context, int qLength)

          { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + }

          +
          + /**
          + * Handle ContainerSchedulerEvents.
          + * @param event ContainerSchedulerEvent.
          + */
          + @Override
          + public void handle(ContainerSchedulerEvent event) {
          + switch (event.getType())

          { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + }

          + }
          +
          + /**
          + * Return number of queued containers.
          + * @return Number of queued containers.
          + */
          + public int getNumQueuedContainers()

          { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedGuaranteedContainers()

          { + return this.queuedGuaranteedContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedOpportunisticContainers()

          { + return this.queuedOpportunisticContainers.size(); + }

          +
          + public OpportunisticContainersStatus getOpportunisticContainersStatus()

          { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + }

          +
          + private void onContainerCompleted(Container container) {
          + // decrement only if it was a running container
          + if (scheduledToRunContainers.containsKey(container.getContainerId())) {
          + this.utilizationManager.subtractContainerResource(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + }

          + }
          + scheduledToRunContainers.remove(container.getContainerId());
          + oppContainersMarkedForKill.remove(container.getContainerId());
          + startPendingContainers();
          + }
          +
          + private void startPendingContainers() {
          + // Start pending guaranteed containers, if resources available.
          + boolean resourcesAvailable =
          + startContainersFromQueue(queuedGuaranteedContainers.values());
          + // Start opportunistic containers, if resources available.
          + if (resourcesAvailable)

          { + startContainersFromQueue(queuedOpportunisticContainers.values()); + }

          + }
          +
          + private boolean startContainersFromQueue(
          + Collection<Container> queuedContainers) {
          + Iterator<Container> cIter = queuedContainers.iterator();
          + boolean resourcesAvailable = true;
          + while (cIter.hasNext() && resourcesAvailable) {
          + Container container = cIter.next();
          + if (this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + cIter.remove(); + }

          else

          { + resourcesAvailable = false; + }

          + }
          + return resourcesAvailable;
          + }
          +
          + @VisibleForTesting
          + protected void scheduleContainer(Container container) {
          + if (maxOppQueueLength <= 0)

          { + startAllocatedContainer(container); + return; + }

          + if (queuedGuaranteedContainers.isEmpty() &&
          + queuedOpportunisticContainers.isEmpty() &&
          + this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + }

          else {
          + LOG.info("No available resources for container {} to start its execution "
          — End diff –

          Should this be at DEBUG level?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87639036 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedOpportunisticContainers = new LinkedHashMap<>(); + + // Used to keep track of containers that have been marked to be killed + // to make room for a guaranteed container. + private final Map<ContainerId, Container> oppContainersMarkedForKill = + new HashMap<>(); + + // Containers launched by the Scheduler will take a while to actually + // move to the RUNNING state, but should still be fair game for killing + // by the scheduler to make room for guaranteed containers. + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers = + new LinkedHashMap<>(); + + private final ContainerQueuingLimit queuingLimit = + ContainerQueuingLimit.newInstance(); + + private final OpportunisticContainersStatus opportunisticContainersStatus; + + // Resource Utilization Manager that decides how utilization of the cluster + // increase / decreases based on container start / finish + private ResourceUtilizationManager utilizationManager; + + /** + * Instantiate a Container Scheduler. + * @param context NodeManager Context. + */ + public ContainerScheduler(Context context) { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + } + + @VisibleForTesting + public ContainerScheduler(Context context, int qLength) { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + } + + /** + * Handle ContainerSchedulerEvents. + * @param event ContainerSchedulerEvent. + */ + @Override + public void handle(ContainerSchedulerEvent event) { + switch (event.getType()) { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + } + } + + /** + * Return number of queued containers. + * @return Number of queued containers. + */ + public int getNumQueuedContainers() { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedGuaranteedContainers() { + return this.queuedGuaranteedContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedOpportunisticContainers() { + return this.queuedOpportunisticContainers.size(); + } + + public OpportunisticContainersStatus getOpportunisticContainersStatus() { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + } + + private void onContainerCompleted(Container container) { + // decrement only if it was a running container + if (scheduledToRunContainers.containsKey(container.getContainerId())) { + this.utilizationManager.subtractContainerResource(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + } + } + scheduledToRunContainers.remove(container.getContainerId()); + oppContainersMarkedForKill.remove(container.getContainerId()); + startPendingContainers(); + } + + private void startPendingContainers() { + // Start pending guaranteed containers, if resources available. + boolean resourcesAvailable = + startContainersFromQueue(queuedGuaranteedContainers.values()); + // Start opportunistic containers, if resources available. + if (resourcesAvailable) { + startContainersFromQueue(queuedOpportunisticContainers.values()); + } + } + + private boolean startContainersFromQueue( + Collection<Container> queuedContainers) { + Iterator<Container> cIter = queuedContainers.iterator(); + boolean resourcesAvailable = true; + while (cIter.hasNext() && resourcesAvailable) { + Container container = cIter.next(); + if (this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + cIter.remove(); + } else { + resourcesAvailable = false; + } + } + return resourcesAvailable; + } + + @VisibleForTesting + protected void scheduleContainer(Container container) { + if (maxOppQueueLength <= 0) { + startAllocatedContainer(container); + return; + } + if (queuedGuaranteedContainers.isEmpty() && + queuedOpportunisticContainers.isEmpty() && + this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + } else { + LOG.info("No available resources for container {} to start its execution " — End diff – Should this be at DEBUG level?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87639946

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          + new HashMap<>();
          +
          + // Containers launched by the Scheduler will take a while to actually
          + // move to the RUNNING state, but should still be fair game for killing
          + // by the scheduler to make room for guaranteed containers.
          + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers =
          + new LinkedHashMap<>();
          +
          + private final ContainerQueuingLimit queuingLimit =
          + ContainerQueuingLimit.newInstance();
          +
          + private final OpportunisticContainersStatus opportunisticContainersStatus;
          +
          + // Resource Utilization Manager that decides how utilization of the cluster
          + // increase / decreases based on container start / finish
          + private ResourceUtilizationManager utilizationManager;
          +
          + /**
          + * Instantiate a Container Scheduler.
          + * @param context NodeManager Context.
          + */
          + public ContainerScheduler(Context context)

          { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + }

          +
          + @VisibleForTesting
          + public ContainerScheduler(Context context, int qLength)

          { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + }

          +
          + /**
          + * Handle ContainerSchedulerEvents.
          + * @param event ContainerSchedulerEvent.
          + */
          + @Override
          + public void handle(ContainerSchedulerEvent event) {
          + switch (event.getType())

          { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + }

          + }
          +
          + /**
          + * Return number of queued containers.
          + * @return Number of queued containers.
          + */
          + public int getNumQueuedContainers()

          { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedGuaranteedContainers()

          { + return this.queuedGuaranteedContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedOpportunisticContainers()

          { + return this.queuedOpportunisticContainers.size(); + }

          +
          + public OpportunisticContainersStatus getOpportunisticContainersStatus()

          { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + }

          +
          + private void onContainerCompleted(Container container) {
          + // decrement only if it was a running container
          + if (scheduledToRunContainers.containsKey(container.getContainerId())) {
          + this.utilizationManager.subtractContainerResource(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + }

          + }
          + scheduledToRunContainers.remove(container.getContainerId());
          + oppContainersMarkedForKill.remove(container.getContainerId());
          + startPendingContainers();
          + }
          +
          + private void startPendingContainers() {
          + // Start pending guaranteed containers, if resources available.
          + boolean resourcesAvailable =
          + startContainersFromQueue(queuedGuaranteedContainers.values());
          + // Start opportunistic containers, if resources available.
          + if (resourcesAvailable)

          { + startContainersFromQueue(queuedOpportunisticContainers.values()); + }

          + }
          +
          + private boolean startContainersFromQueue(
          + Collection<Container> queuedContainers) {
          + Iterator<Container> cIter = queuedContainers.iterator();
          + boolean resourcesAvailable = true;
          + while (cIter.hasNext() && resourcesAvailable) {
          + Container container = cIter.next();
          + if (this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + cIter.remove(); + }

          else

          { + resourcesAvailable = false; + }

          + }
          + return resourcesAvailable;
          + }
          +
          + @VisibleForTesting
          + protected void scheduleContainer(Container container) {
          + if (maxOppQueueLength <= 0)

          { + startAllocatedContainer(container); + return; + }

          + if (queuedGuaranteedContainers.isEmpty() &&
          + queuedOpportunisticContainers.isEmpty() &&
          + this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + }

          else {
          + LOG.info("No available resources for container {} to start its execution "
          + + "immediately.", container.getContainerId());
          + boolean isQueued = true;
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.GUARANTEED)

          { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + }

          else {
          + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) {
          + LOG.info("Opportunistic container {} will be queued at the NM.",
          + container.getContainerId());
          + queuedOpportunisticContainers.put(
          + container.getContainerId(), container);
          + } else {
          + isQueued = false;
          + LOG.info("Opportunistic container [{}] will not be queued at the NM" +
          + "since max queue length [{}] has been reached",
          + container.getContainerId(), maxOppQueueLength);
          + container.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Opportunistic container queue is full.");
          + }
          + }
          + if (isQueued) {
          + try

          { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + }

          catch (IOException e)

          { + LOG.warn("Could not store container state into store..", e); + }

          + }
          + }
          + }
          +
          + private void killOpportunisticContainers(Container container) {
          + List<Container> extraOpportContainersToKill =
          + pickOpportunisticContainersToKill(container.getContainerId());
          + // Kill the opportunistic containers that were chosen.
          + for (Container contToKill : extraOpportContainersToKill) {
          + contToKill.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Container Killed to make room for Guaranteed Container.");
          + oppContainersMarkedForKill.put(contToKill.getContainerId(), contToKill);
          + LOG.info(
          + "Opportunistic container {} will be killed in order to start the "
          + + "execution of guaranteed container {}.",
          + contToKill.getContainerId(), container.getContainerId());
          + }
          + }
          +
          + private void startAllocatedContainer(Container container) {
          + LOG.info("Starting container [" + container.getContainerId()+ "]");
          + scheduledToRunContainers.put(container.getContainerId(), container);
          + this.utilizationManager.addContainerResources(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + + container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + + container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + + 1); + }

          + container.sendLaunchEvent();
          + }
          +
          + private List<Container> pickOpportunisticContainersToKill(
          + ContainerId containerToStartId) {
          + // The additional opportunistic containers that need to be killed for the
          — End diff –

          Again, why are these additional?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87639946 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedOpportunisticContainers = new LinkedHashMap<>(); + + // Used to keep track of containers that have been marked to be killed + // to make room for a guaranteed container. + private final Map<ContainerId, Container> oppContainersMarkedForKill = + new HashMap<>(); + + // Containers launched by the Scheduler will take a while to actually + // move to the RUNNING state, but should still be fair game for killing + // by the scheduler to make room for guaranteed containers. + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers = + new LinkedHashMap<>(); + + private final ContainerQueuingLimit queuingLimit = + ContainerQueuingLimit.newInstance(); + + private final OpportunisticContainersStatus opportunisticContainersStatus; + + // Resource Utilization Manager that decides how utilization of the cluster + // increase / decreases based on container start / finish + private ResourceUtilizationManager utilizationManager; + + /** + * Instantiate a Container Scheduler. + * @param context NodeManager Context. + */ + public ContainerScheduler(Context context) { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + } + + @VisibleForTesting + public ContainerScheduler(Context context, int qLength) { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + } + + /** + * Handle ContainerSchedulerEvents. + * @param event ContainerSchedulerEvent. + */ + @Override + public void handle(ContainerSchedulerEvent event) { + switch (event.getType()) { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + } + } + + /** + * Return number of queued containers. + * @return Number of queued containers. + */ + public int getNumQueuedContainers() { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedGuaranteedContainers() { + return this.queuedGuaranteedContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedOpportunisticContainers() { + return this.queuedOpportunisticContainers.size(); + } + + public OpportunisticContainersStatus getOpportunisticContainersStatus() { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + } + + private void onContainerCompleted(Container container) { + // decrement only if it was a running container + if (scheduledToRunContainers.containsKey(container.getContainerId())) { + this.utilizationManager.subtractContainerResource(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + } + } + scheduledToRunContainers.remove(container.getContainerId()); + oppContainersMarkedForKill.remove(container.getContainerId()); + startPendingContainers(); + } + + private void startPendingContainers() { + // Start pending guaranteed containers, if resources available. + boolean resourcesAvailable = + startContainersFromQueue(queuedGuaranteedContainers.values()); + // Start opportunistic containers, if resources available. + if (resourcesAvailable) { + startContainersFromQueue(queuedOpportunisticContainers.values()); + } + } + + private boolean startContainersFromQueue( + Collection<Container> queuedContainers) { + Iterator<Container> cIter = queuedContainers.iterator(); + boolean resourcesAvailable = true; + while (cIter.hasNext() && resourcesAvailable) { + Container container = cIter.next(); + if (this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + cIter.remove(); + } else { + resourcesAvailable = false; + } + } + return resourcesAvailable; + } + + @VisibleForTesting + protected void scheduleContainer(Container container) { + if (maxOppQueueLength <= 0) { + startAllocatedContainer(container); + return; + } + if (queuedGuaranteedContainers.isEmpty() && + queuedOpportunisticContainers.isEmpty() && + this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + } else { + LOG.info("No available resources for container {} to start its execution " + + "immediately.", container.getContainerId()); + boolean isQueued = true; + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.GUARANTEED) { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + } else { + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) { + LOG.info("Opportunistic container {} will be queued at the NM.", + container.getContainerId()); + queuedOpportunisticContainers.put( + container.getContainerId(), container); + } else { + isQueued = false; + LOG.info("Opportunistic container [{}] will not be queued at the NM" + + "since max queue length [{}] has been reached", + container.getContainerId(), maxOppQueueLength); + container.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Opportunistic container queue is full."); + } + } + if (isQueued) { + try { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + } catch (IOException e) { + LOG.warn("Could not store container state into store..", e); + } + } + } + } + + private void killOpportunisticContainers(Container container) { + List<Container> extraOpportContainersToKill = + pickOpportunisticContainersToKill(container.getContainerId()); + // Kill the opportunistic containers that were chosen. + for (Container contToKill : extraOpportContainersToKill) { + contToKill.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Container Killed to make room for Guaranteed Container."); + oppContainersMarkedForKill.put(contToKill.getContainerId(), contToKill); + LOG.info( + "Opportunistic container {} will be killed in order to start the " + + "execution of guaranteed container {}.", + contToKill.getContainerId(), container.getContainerId()); + } + } + + private void startAllocatedContainer(Container container) { + LOG.info("Starting container [" + container.getContainerId()+ "] "); + scheduledToRunContainers.put(container.getContainerId(), container); + this.utilizationManager.addContainerResources(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + + container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + + container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + + 1); + } + container.sendLaunchEvent(); + } + + private List<Container> pickOpportunisticContainersToKill( + ContainerId containerToStartId) { + // The additional opportunistic containers that need to be killed for the — End diff – Again, why are these additional?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87639879

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          + new HashMap<>();
          +
          + // Containers launched by the Scheduler will take a while to actually
          + // move to the RUNNING state, but should still be fair game for killing
          + // by the scheduler to make room for guaranteed containers.
          + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers =
          — End diff –

          Also, should this be a stack? So, containers can be killed in a LIFO manner?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87639879 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedOpportunisticContainers = new LinkedHashMap<>(); + + // Used to keep track of containers that have been marked to be killed + // to make room for a guaranteed container. + private final Map<ContainerId, Container> oppContainersMarkedForKill = + new HashMap<>(); + + // Containers launched by the Scheduler will take a while to actually + // move to the RUNNING state, but should still be fair game for killing + // by the scheduler to make room for guaranteed containers. + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers = — End diff – Also, should this be a stack? So, containers can be killed in a LIFO manner?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87639757

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          + new HashMap<>();
          +
          + // Containers launched by the Scheduler will take a while to actually
          + // move to the RUNNING state, but should still be fair game for killing
          + // by the scheduler to make room for guaranteed containers.
          + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers =
          + new LinkedHashMap<>();
          +
          + private final ContainerQueuingLimit queuingLimit =
          + ContainerQueuingLimit.newInstance();
          +
          + private final OpportunisticContainersStatus opportunisticContainersStatus;
          +
          + // Resource Utilization Manager that decides how utilization of the cluster
          + // increase / decreases based on container start / finish
          + private ResourceUtilizationManager utilizationManager;
          +
          + /**
          + * Instantiate a Container Scheduler.
          + * @param context NodeManager Context.
          + */
          + public ContainerScheduler(Context context)

          { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + }

          +
          + @VisibleForTesting
          + public ContainerScheduler(Context context, int qLength)

          { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + }

          +
          + /**
          + * Handle ContainerSchedulerEvents.
          + * @param event ContainerSchedulerEvent.
          + */
          + @Override
          + public void handle(ContainerSchedulerEvent event) {
          + switch (event.getType())

          { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + }

          + }
          +
          + /**
          + * Return number of queued containers.
          + * @return Number of queued containers.
          + */
          + public int getNumQueuedContainers()

          { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedGuaranteedContainers()

          { + return this.queuedGuaranteedContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedOpportunisticContainers()

          { + return this.queuedOpportunisticContainers.size(); + }

          +
          + public OpportunisticContainersStatus getOpportunisticContainersStatus()

          { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + }

          +
          + private void onContainerCompleted(Container container) {
          + // decrement only if it was a running container
          + if (scheduledToRunContainers.containsKey(container.getContainerId())) {
          + this.utilizationManager.subtractContainerResource(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + }

          + }
          + scheduledToRunContainers.remove(container.getContainerId());
          + oppContainersMarkedForKill.remove(container.getContainerId());
          + startPendingContainers();
          + }
          +
          + private void startPendingContainers() {
          + // Start pending guaranteed containers, if resources available.
          + boolean resourcesAvailable =
          + startContainersFromQueue(queuedGuaranteedContainers.values());
          + // Start opportunistic containers, if resources available.
          + if (resourcesAvailable)

          { + startContainersFromQueue(queuedOpportunisticContainers.values()); + }

          + }
          +
          + private boolean startContainersFromQueue(
          + Collection<Container> queuedContainers) {
          + Iterator<Container> cIter = queuedContainers.iterator();
          + boolean resourcesAvailable = true;
          + while (cIter.hasNext() && resourcesAvailable) {
          + Container container = cIter.next();
          + if (this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + cIter.remove(); + }

          else

          { + resourcesAvailable = false; + }

          + }
          + return resourcesAvailable;
          + }
          +
          + @VisibleForTesting
          + protected void scheduleContainer(Container container) {
          + if (maxOppQueueLength <= 0)

          { + startAllocatedContainer(container); + return; + }

          + if (queuedGuaranteedContainers.isEmpty() &&
          + queuedOpportunisticContainers.isEmpty() &&
          + this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + }

          else {
          + LOG.info("No available resources for container {} to start its execution "
          + + "immediately.", container.getContainerId());
          + boolean isQueued = true;
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.GUARANTEED)

          { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + }

          else {
          + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) {
          + LOG.info("Opportunistic container {} will be queued at the NM.",
          + container.getContainerId());
          + queuedOpportunisticContainers.put(
          + container.getContainerId(), container);
          + } else {
          + isQueued = false;
          + LOG.info("Opportunistic container [{}] will not be queued at the NM" +
          + "since max queue length [{}] has been reached",
          + container.getContainerId(), maxOppQueueLength);
          + container.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Opportunistic container queue is full.");
          + }
          + }
          + if (isQueued) {
          + try

          { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + }

          catch (IOException e)

          { + LOG.warn("Could not store container state into store..", e); + }

          + }
          + }
          + }
          +
          + private void killOpportunisticContainers(Container container) {
          + List<Container> extraOpportContainersToKill =
          + pickOpportunisticContainersToKill(container.getContainerId());
          + // Kill the opportunistic containers that were chosen.
          + for (Container contToKill : extraOpportContainersToKill) {
          + contToKill.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Container Killed to make room for Guaranteed Container.");
          + oppContainersMarkedForKill.put(contToKill.getContainerId(), contToKill);
          + LOG.info(
          + "Opportunistic container {} will be killed in order to start the "
          + + "execution of guaranteed container {}.",
          + contToKill.getContainerId(), container.getContainerId());
          + }
          + }
          +
          + private void startAllocatedContainer(Container container) {
          + LOG.info("Starting container [" + container.getContainerId()+ "]");
          + scheduledToRunContainers.put(container.getContainerId(), container);
          + this.utilizationManager.addContainerResources(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC) {
          + this.opportunisticContainersStatus.setOpportMemoryUsed(
          — End diff –

          Method name has "used", while we actually refer to allocated. Should we name this differently?

          Or, would it make more sense to add another boolean to OpportunisticContainersStatus to capture whether used refers to allocated/utilization?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87639757 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedOpportunisticContainers = new LinkedHashMap<>(); + + // Used to keep track of containers that have been marked to be killed + // to make room for a guaranteed container. + private final Map<ContainerId, Container> oppContainersMarkedForKill = + new HashMap<>(); + + // Containers launched by the Scheduler will take a while to actually + // move to the RUNNING state, but should still be fair game for killing + // by the scheduler to make room for guaranteed containers. + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers = + new LinkedHashMap<>(); + + private final ContainerQueuingLimit queuingLimit = + ContainerQueuingLimit.newInstance(); + + private final OpportunisticContainersStatus opportunisticContainersStatus; + + // Resource Utilization Manager that decides how utilization of the cluster + // increase / decreases based on container start / finish + private ResourceUtilizationManager utilizationManager; + + /** + * Instantiate a Container Scheduler. + * @param context NodeManager Context. + */ + public ContainerScheduler(Context context) { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + } + + @VisibleForTesting + public ContainerScheduler(Context context, int qLength) { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + } + + /** + * Handle ContainerSchedulerEvents. + * @param event ContainerSchedulerEvent. + */ + @Override + public void handle(ContainerSchedulerEvent event) { + switch (event.getType()) { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + } + } + + /** + * Return number of queued containers. + * @return Number of queued containers. + */ + public int getNumQueuedContainers() { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedGuaranteedContainers() { + return this.queuedGuaranteedContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedOpportunisticContainers() { + return this.queuedOpportunisticContainers.size(); + } + + public OpportunisticContainersStatus getOpportunisticContainersStatus() { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + } + + private void onContainerCompleted(Container container) { + // decrement only if it was a running container + if (scheduledToRunContainers.containsKey(container.getContainerId())) { + this.utilizationManager.subtractContainerResource(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + } + } + scheduledToRunContainers.remove(container.getContainerId()); + oppContainersMarkedForKill.remove(container.getContainerId()); + startPendingContainers(); + } + + private void startPendingContainers() { + // Start pending guaranteed containers, if resources available. + boolean resourcesAvailable = + startContainersFromQueue(queuedGuaranteedContainers.values()); + // Start opportunistic containers, if resources available. + if (resourcesAvailable) { + startContainersFromQueue(queuedOpportunisticContainers.values()); + } + } + + private boolean startContainersFromQueue( + Collection<Container> queuedContainers) { + Iterator<Container> cIter = queuedContainers.iterator(); + boolean resourcesAvailable = true; + while (cIter.hasNext() && resourcesAvailable) { + Container container = cIter.next(); + if (this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + cIter.remove(); + } else { + resourcesAvailable = false; + } + } + return resourcesAvailable; + } + + @VisibleForTesting + protected void scheduleContainer(Container container) { + if (maxOppQueueLength <= 0) { + startAllocatedContainer(container); + return; + } + if (queuedGuaranteedContainers.isEmpty() && + queuedOpportunisticContainers.isEmpty() && + this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + } else { + LOG.info("No available resources for container {} to start its execution " + + "immediately.", container.getContainerId()); + boolean isQueued = true; + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.GUARANTEED) { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + } else { + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) { + LOG.info("Opportunistic container {} will be queued at the NM.", + container.getContainerId()); + queuedOpportunisticContainers.put( + container.getContainerId(), container); + } else { + isQueued = false; + LOG.info("Opportunistic container [{}] will not be queued at the NM" + + "since max queue length [{}] has been reached", + container.getContainerId(), maxOppQueueLength); + container.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Opportunistic container queue is full."); + } + } + if (isQueued) { + try { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + } catch (IOException e) { + LOG.warn("Could not store container state into store..", e); + } + } + } + } + + private void killOpportunisticContainers(Container container) { + List<Container> extraOpportContainersToKill = + pickOpportunisticContainersToKill(container.getContainerId()); + // Kill the opportunistic containers that were chosen. + for (Container contToKill : extraOpportContainersToKill) { + contToKill.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Container Killed to make room for Guaranteed Container."); + oppContainersMarkedForKill.put(contToKill.getContainerId(), contToKill); + LOG.info( + "Opportunistic container {} will be killed in order to start the " + + "execution of guaranteed container {}.", + contToKill.getContainerId(), container.getContainerId()); + } + } + + private void startAllocatedContainer(Container container) { + LOG.info("Starting container [" + container.getContainerId()+ "] "); + scheduledToRunContainers.put(container.getContainerId(), container); + this.utilizationManager.addContainerResources(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( — End diff – Method name has "used", while we actually refer to allocated. Should we name this differently? Or, would it make more sense to add another boolean to OpportunisticContainersStatus to capture whether used refers to allocated/utilization?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87639223

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          + new HashMap<>();
          +
          + // Containers launched by the Scheduler will take a while to actually
          + // move to the RUNNING state, but should still be fair game for killing
          + // by the scheduler to make room for guaranteed containers.
          + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers =
          + new LinkedHashMap<>();
          +
          + private final ContainerQueuingLimit queuingLimit =
          + ContainerQueuingLimit.newInstance();
          +
          + private final OpportunisticContainersStatus opportunisticContainersStatus;
          +
          + // Resource Utilization Manager that decides how utilization of the cluster
          + // increase / decreases based on container start / finish
          + private ResourceUtilizationManager utilizationManager;
          +
          + /**
          + * Instantiate a Container Scheduler.
          + * @param context NodeManager Context.
          + */
          + public ContainerScheduler(Context context)

          { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + }

          +
          + @VisibleForTesting
          + public ContainerScheduler(Context context, int qLength)

          { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + }

          +
          + /**
          + * Handle ContainerSchedulerEvents.
          + * @param event ContainerSchedulerEvent.
          + */
          + @Override
          + public void handle(ContainerSchedulerEvent event) {
          + switch (event.getType())

          { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + }

          + }
          +
          + /**
          + * Return number of queued containers.
          + * @return Number of queued containers.
          + */
          + public int getNumQueuedContainers()

          { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedGuaranteedContainers()

          { + return this.queuedGuaranteedContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedOpportunisticContainers()

          { + return this.queuedOpportunisticContainers.size(); + }

          +
          + public OpportunisticContainersStatus getOpportunisticContainersStatus()

          { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + }

          +
          + private void onContainerCompleted(Container container) {
          + // decrement only if it was a running container
          + if (scheduledToRunContainers.containsKey(container.getContainerId())) {
          + this.utilizationManager.subtractContainerResource(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + }

          + }
          + scheduledToRunContainers.remove(container.getContainerId());
          + oppContainersMarkedForKill.remove(container.getContainerId());
          + startPendingContainers();
          + }
          +
          + private void startPendingContainers() {
          + // Start pending guaranteed containers, if resources available.
          + boolean resourcesAvailable =
          + startContainersFromQueue(queuedGuaranteedContainers.values());
          + // Start opportunistic containers, if resources available.
          + if (resourcesAvailable)

          { + startContainersFromQueue(queuedOpportunisticContainers.values()); + }

          + }
          +
          + private boolean startContainersFromQueue(
          + Collection<Container> queuedContainers) {
          + Iterator<Container> cIter = queuedContainers.iterator();
          + boolean resourcesAvailable = true;
          + while (cIter.hasNext() && resourcesAvailable) {
          + Container container = cIter.next();
          + if (this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + cIter.remove(); + }

          else

          { + resourcesAvailable = false; + }

          + }
          + return resourcesAvailable;
          + }
          +
          + @VisibleForTesting
          + protected void scheduleContainer(Container container) {
          + if (maxOppQueueLength <= 0)

          { + startAllocatedContainer(container); + return; + }

          + if (queuedGuaranteedContainers.isEmpty() &&
          + queuedOpportunisticContainers.isEmpty() &&
          + this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + }

          else {
          + LOG.info("No available resources for container {} to start its execution "
          + + "immediately.", container.getContainerId());
          + boolean isQueued = true;
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.GUARANTEED)

          { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + }

          else {
          + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) {
          + LOG.info("Opportunistic container {} will be queued at the NM.",
          + container.getContainerId());
          + queuedOpportunisticContainers.put(
          + container.getContainerId(), container);
          + } else {
          + isQueued = false;
          + LOG.info("Opportunistic container [{}] will not be queued at the NM" +
          + "since max queue length [{}] has been reached",
          + container.getContainerId(), maxOppQueueLength);
          + container.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Opportunistic container queue is full.");
          + }
          + }
          + if (isQueued) {
          + try

          { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + }

          catch (IOException e)

          { + LOG.warn("Could not store container state into store..", e); + }

          + }
          + }
          + }
          +
          + private void killOpportunisticContainers(Container container) {
          + List<Container> extraOpportContainersToKill =
          — End diff –

          Why is this extra? What is the base?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87639223 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedOpportunisticContainers = new LinkedHashMap<>(); + + // Used to keep track of containers that have been marked to be killed + // to make room for a guaranteed container. + private final Map<ContainerId, Container> oppContainersMarkedForKill = + new HashMap<>(); + + // Containers launched by the Scheduler will take a while to actually + // move to the RUNNING state, but should still be fair game for killing + // by the scheduler to make room for guaranteed containers. + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers = + new LinkedHashMap<>(); + + private final ContainerQueuingLimit queuingLimit = + ContainerQueuingLimit.newInstance(); + + private final OpportunisticContainersStatus opportunisticContainersStatus; + + // Resource Utilization Manager that decides how utilization of the cluster + // increase / decreases based on container start / finish + private ResourceUtilizationManager utilizationManager; + + /** + * Instantiate a Container Scheduler. + * @param context NodeManager Context. + */ + public ContainerScheduler(Context context) { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + } + + @VisibleForTesting + public ContainerScheduler(Context context, int qLength) { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + } + + /** + * Handle ContainerSchedulerEvents. + * @param event ContainerSchedulerEvent. + */ + @Override + public void handle(ContainerSchedulerEvent event) { + switch (event.getType()) { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + } + } + + /** + * Return number of queued containers. + * @return Number of queued containers. + */ + public int getNumQueuedContainers() { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedGuaranteedContainers() { + return this.queuedGuaranteedContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedOpportunisticContainers() { + return this.queuedOpportunisticContainers.size(); + } + + public OpportunisticContainersStatus getOpportunisticContainersStatus() { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + } + + private void onContainerCompleted(Container container) { + // decrement only if it was a running container + if (scheduledToRunContainers.containsKey(container.getContainerId())) { + this.utilizationManager.subtractContainerResource(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + } + } + scheduledToRunContainers.remove(container.getContainerId()); + oppContainersMarkedForKill.remove(container.getContainerId()); + startPendingContainers(); + } + + private void startPendingContainers() { + // Start pending guaranteed containers, if resources available. + boolean resourcesAvailable = + startContainersFromQueue(queuedGuaranteedContainers.values()); + // Start opportunistic containers, if resources available. + if (resourcesAvailable) { + startContainersFromQueue(queuedOpportunisticContainers.values()); + } + } + + private boolean startContainersFromQueue( + Collection<Container> queuedContainers) { + Iterator<Container> cIter = queuedContainers.iterator(); + boolean resourcesAvailable = true; + while (cIter.hasNext() && resourcesAvailable) { + Container container = cIter.next(); + if (this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + cIter.remove(); + } else { + resourcesAvailable = false; + } + } + return resourcesAvailable; + } + + @VisibleForTesting + protected void scheduleContainer(Container container) { + if (maxOppQueueLength <= 0) { + startAllocatedContainer(container); + return; + } + if (queuedGuaranteedContainers.isEmpty() && + queuedOpportunisticContainers.isEmpty() && + this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + } else { + LOG.info("No available resources for container {} to start its execution " + + "immediately.", container.getContainerId()); + boolean isQueued = true; + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.GUARANTEED) { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + } else { + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) { + LOG.info("Opportunistic container {} will be queued at the NM.", + container.getContainerId()); + queuedOpportunisticContainers.put( + container.getContainerId(), container); + } else { + isQueued = false; + LOG.info("Opportunistic container [{}] will not be queued at the NM" + + "since max queue length [{}] has been reached", + container.getContainerId(), maxOppQueueLength); + container.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Opportunistic container queue is full."); + } + } + if (isQueued) { + try { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + } catch (IOException e) { + LOG.warn("Could not store container state into store..", e); + } + } + } + } + + private void killOpportunisticContainers(Container container) { + List<Container> extraOpportContainersToKill = — End diff – Why is this extra? What is the base?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87640635

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          — End diff –

          Same comment about it being a queue here..

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87640635 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> — End diff – Same comment about it being a queue here..
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87640580

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          — End diff –

          Why do these need to be a map, why not a queue? Is it because there could be multiple Container objects corresponding to one ContainerId?

          If yes, would it make sense to implement equals and hashCode on Container to ensure we check only ContainerId?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87640580 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> — End diff – Why do these need to be a map, why not a queue? Is it because there could be multiple Container objects corresponding to one ContainerId? If yes, would it make sense to implement equals and hashCode on Container to ensure we check only ContainerId?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87640015

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          + new HashMap<>();
          +
          + // Containers launched by the Scheduler will take a while to actually
          + // move to the RUNNING state, but should still be fair game for killing
          + // by the scheduler to make room for guaranteed containers.
          + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers =
          + new LinkedHashMap<>();
          +
          + private final ContainerQueuingLimit queuingLimit =
          + ContainerQueuingLimit.newInstance();
          +
          + private final OpportunisticContainersStatus opportunisticContainersStatus;
          +
          + // Resource Utilization Manager that decides how utilization of the cluster
          + // increase / decreases based on container start / finish
          + private ResourceUtilizationManager utilizationManager;
          +
          + /**
          + * Instantiate a Container Scheduler.
          + * @param context NodeManager Context.
          + */
          + public ContainerScheduler(Context context)

          { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + }

          +
          + @VisibleForTesting
          + public ContainerScheduler(Context context, int qLength)

          { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + }

          +
          + /**
          + * Handle ContainerSchedulerEvents.
          + * @param event ContainerSchedulerEvent.
          + */
          + @Override
          + public void handle(ContainerSchedulerEvent event) {
          + switch (event.getType())

          { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + }

          + }
          +
          + /**
          + * Return number of queued containers.
          + * @return Number of queued containers.
          + */
          + public int getNumQueuedContainers()

          { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedGuaranteedContainers()

          { + return this.queuedGuaranteedContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedOpportunisticContainers()

          { + return this.queuedOpportunisticContainers.size(); + }

          +
          + public OpportunisticContainersStatus getOpportunisticContainersStatus()

          { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + }

          +
          + private void onContainerCompleted(Container container) {
          + // decrement only if it was a running container
          + if (scheduledToRunContainers.containsKey(container.getContainerId())) {
          + this.utilizationManager.subtractContainerResource(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + }

          + }
          + scheduledToRunContainers.remove(container.getContainerId());
          + oppContainersMarkedForKill.remove(container.getContainerId());
          + startPendingContainers();
          + }
          +
          + private void startPendingContainers() {
          + // Start pending guaranteed containers, if resources available.
          + boolean resourcesAvailable =
          + startContainersFromQueue(queuedGuaranteedContainers.values());
          + // Start opportunistic containers, if resources available.
          + if (resourcesAvailable)

          { + startContainersFromQueue(queuedOpportunisticContainers.values()); + }

          + }
          +
          + private boolean startContainersFromQueue(
          + Collection<Container> queuedContainers) {
          + Iterator<Container> cIter = queuedContainers.iterator();
          + boolean resourcesAvailable = true;
          + while (cIter.hasNext() && resourcesAvailable) {
          + Container container = cIter.next();
          + if (this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + cIter.remove(); + }

          else

          { + resourcesAvailable = false; + }

          + }
          + return resourcesAvailable;
          + }
          +
          + @VisibleForTesting
          + protected void scheduleContainer(Container container) {
          + if (maxOppQueueLength <= 0)

          { + startAllocatedContainer(container); + return; + }

          + if (queuedGuaranteedContainers.isEmpty() &&
          + queuedOpportunisticContainers.isEmpty() &&
          + this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + }

          else {
          + LOG.info("No available resources for container {} to start its execution "
          + + "immediately.", container.getContainerId());
          + boolean isQueued = true;
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.GUARANTEED)

          { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + }

          else {
          + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) {
          + LOG.info("Opportunistic container {} will be queued at the NM.",
          + container.getContainerId());
          + queuedOpportunisticContainers.put(
          + container.getContainerId(), container);
          + } else {
          + isQueued = false;
          + LOG.info("Opportunistic container [{}] will not be queued at the NM" +
          + "since max queue length [{}] has been reached",
          + container.getContainerId(), maxOppQueueLength);
          + container.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Opportunistic container queue is full.");
          + }
          + }
          + if (isQueued) {
          + try

          { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + }

          catch (IOException e)

          { + LOG.warn("Could not store container state into store..", e); + }

          + }
          + }
          + }
          +
          + private void killOpportunisticContainers(Container container) {
          + List<Container> extraOpportContainersToKill =
          + pickOpportunisticContainersToKill(container.getContainerId());
          + // Kill the opportunistic containers that were chosen.
          + for (Container contToKill : extraOpportContainersToKill) {
          + contToKill.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Container Killed to make room for Guaranteed Container.");
          + oppContainersMarkedForKill.put(contToKill.getContainerId(), contToKill);
          + LOG.info(
          + "Opportunistic container {} will be killed in order to start the "
          + + "execution of guaranteed container {}.",
          + contToKill.getContainerId(), container.getContainerId());
          + }
          + }
          +
          + private void startAllocatedContainer(Container container) {
          + LOG.info("Starting container [" + container.getContainerId()+ "]");
          + scheduledToRunContainers.put(container.getContainerId(), container);
          + this.utilizationManager.addContainerResources(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + + container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + + container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + + 1); + }

          + container.sendLaunchEvent();
          + }
          +
          + private List<Container> pickOpportunisticContainersToKill(
          + ContainerId containerToStartId) {
          + // The additional opportunistic containers that need to be killed for the
          + // given container to start.
          + List<Container> extraOpportContainersToKill = new ArrayList<>();
          — End diff –

          Same comment about extra...

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87640015 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedOpportunisticContainers = new LinkedHashMap<>(); + + // Used to keep track of containers that have been marked to be killed + // to make room for a guaranteed container. + private final Map<ContainerId, Container> oppContainersMarkedForKill = + new HashMap<>(); + + // Containers launched by the Scheduler will take a while to actually + // move to the RUNNING state, but should still be fair game for killing + // by the scheduler to make room for guaranteed containers. + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers = + new LinkedHashMap<>(); + + private final ContainerQueuingLimit queuingLimit = + ContainerQueuingLimit.newInstance(); + + private final OpportunisticContainersStatus opportunisticContainersStatus; + + // Resource Utilization Manager that decides how utilization of the cluster + // increase / decreases based on container start / finish + private ResourceUtilizationManager utilizationManager; + + /** + * Instantiate a Container Scheduler. + * @param context NodeManager Context. + */ + public ContainerScheduler(Context context) { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + } + + @VisibleForTesting + public ContainerScheduler(Context context, int qLength) { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + } + + /** + * Handle ContainerSchedulerEvents. + * @param event ContainerSchedulerEvent. + */ + @Override + public void handle(ContainerSchedulerEvent event) { + switch (event.getType()) { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + } + } + + /** + * Return number of queued containers. + * @return Number of queued containers. + */ + public int getNumQueuedContainers() { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedGuaranteedContainers() { + return this.queuedGuaranteedContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedOpportunisticContainers() { + return this.queuedOpportunisticContainers.size(); + } + + public OpportunisticContainersStatus getOpportunisticContainersStatus() { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + } + + private void onContainerCompleted(Container container) { + // decrement only if it was a running container + if (scheduledToRunContainers.containsKey(container.getContainerId())) { + this.utilizationManager.subtractContainerResource(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + } + } + scheduledToRunContainers.remove(container.getContainerId()); + oppContainersMarkedForKill.remove(container.getContainerId()); + startPendingContainers(); + } + + private void startPendingContainers() { + // Start pending guaranteed containers, if resources available. + boolean resourcesAvailable = + startContainersFromQueue(queuedGuaranteedContainers.values()); + // Start opportunistic containers, if resources available. + if (resourcesAvailable) { + startContainersFromQueue(queuedOpportunisticContainers.values()); + } + } + + private boolean startContainersFromQueue( + Collection<Container> queuedContainers) { + Iterator<Container> cIter = queuedContainers.iterator(); + boolean resourcesAvailable = true; + while (cIter.hasNext() && resourcesAvailable) { + Container container = cIter.next(); + if (this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + cIter.remove(); + } else { + resourcesAvailable = false; + } + } + return resourcesAvailable; + } + + @VisibleForTesting + protected void scheduleContainer(Container container) { + if (maxOppQueueLength <= 0) { + startAllocatedContainer(container); + return; + } + if (queuedGuaranteedContainers.isEmpty() && + queuedOpportunisticContainers.isEmpty() && + this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + } else { + LOG.info("No available resources for container {} to start its execution " + + "immediately.", container.getContainerId()); + boolean isQueued = true; + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.GUARANTEED) { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + } else { + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) { + LOG.info("Opportunistic container {} will be queued at the NM.", + container.getContainerId()); + queuedOpportunisticContainers.put( + container.getContainerId(), container); + } else { + isQueued = false; + LOG.info("Opportunistic container [{}] will not be queued at the NM" + + "since max queue length [{}] has been reached", + container.getContainerId(), maxOppQueueLength); + container.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Opportunistic container queue is full."); + } + } + if (isQueued) { + try { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + } catch (IOException e) { + LOG.warn("Could not store container state into store..", e); + } + } + } + } + + private void killOpportunisticContainers(Container container) { + List<Container> extraOpportContainersToKill = + pickOpportunisticContainersToKill(container.getContainerId()); + // Kill the opportunistic containers that were chosen. + for (Container contToKill : extraOpportContainersToKill) { + contToKill.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Container Killed to make room for Guaranteed Container."); + oppContainersMarkedForKill.put(contToKill.getContainerId(), contToKill); + LOG.info( + "Opportunistic container {} will be killed in order to start the " + + "execution of guaranteed container {}.", + contToKill.getContainerId(), container.getContainerId()); + } + } + + private void startAllocatedContainer(Container container) { + LOG.info("Starting container [" + container.getContainerId()+ "] "); + scheduledToRunContainers.put(container.getContainerId(), container); + this.utilizationManager.addContainerResources(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + + container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + + container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + + 1); + } + container.sendLaunchEvent(); + } + + private List<Container> pickOpportunisticContainersToKill( + ContainerId containerToStartId) { + // The additional opportunistic containers that need to be killed for the + // given container to start. + List<Container> extraOpportContainersToKill = new ArrayList<>(); — End diff – Same comment about extra...
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87641557

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          + new HashMap<>();
          +
          + // Containers launched by the Scheduler will take a while to actually
          + // move to the RUNNING state, but should still be fair game for killing
          + // by the scheduler to make room for guaranteed containers.
          + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers =
          + new LinkedHashMap<>();
          +
          + private final ContainerQueuingLimit queuingLimit =
          + ContainerQueuingLimit.newInstance();
          +
          + private final OpportunisticContainersStatus opportunisticContainersStatus;
          +
          + // Resource Utilization Manager that decides how utilization of the cluster
          + // increase / decreases based on container start / finish
          + private ResourceUtilizationManager utilizationManager;
          +
          + /**
          + * Instantiate a Container Scheduler.
          + * @param context NodeManager Context.
          + */
          + public ContainerScheduler(Context context)

          { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + }

          +
          + @VisibleForTesting
          + public ContainerScheduler(Context context, int qLength)

          { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + }

          +
          + /**
          + * Handle ContainerSchedulerEvents.
          + * @param event ContainerSchedulerEvent.
          + */
          + @Override
          + public void handle(ContainerSchedulerEvent event) {
          + switch (event.getType())

          { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + }

          + }
          +
          + /**
          + * Return number of queued containers.
          + * @return Number of queued containers.
          + */
          + public int getNumQueuedContainers()

          { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedGuaranteedContainers()

          { + return this.queuedGuaranteedContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedOpportunisticContainers()

          { + return this.queuedOpportunisticContainers.size(); + }

          +
          + public OpportunisticContainersStatus getOpportunisticContainersStatus()

          { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + }

          +
          + private void onContainerCompleted(Container container) {
          + // decrement only if it was a running container
          + if (scheduledToRunContainers.containsKey(container.getContainerId())) {
          + this.utilizationManager.subtractContainerResource(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + }

          + }
          + scheduledToRunContainers.remove(container.getContainerId());
          + oppContainersMarkedForKill.remove(container.getContainerId());
          + startPendingContainers();
          + }
          +
          + private void startPendingContainers() {
          + // Start pending guaranteed containers, if resources available.
          + boolean resourcesAvailable =
          + startContainersFromQueue(queuedGuaranteedContainers.values());
          + // Start opportunistic containers, if resources available.
          + if (resourcesAvailable)

          { + startContainersFromQueue(queuedOpportunisticContainers.values()); + }

          + }
          +
          + private boolean startContainersFromQueue(
          + Collection<Container> queuedContainers) {
          + Iterator<Container> cIter = queuedContainers.iterator();
          + boolean resourcesAvailable = true;
          + while (cIter.hasNext() && resourcesAvailable) {
          + Container container = cIter.next();
          + if (this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + cIter.remove(); + }

          else

          { + resourcesAvailable = false; + }

          + }
          + return resourcesAvailable;
          + }
          +
          + @VisibleForTesting
          + protected void scheduleContainer(Container container) {
          + if (maxOppQueueLength <= 0)

          { + startAllocatedContainer(container); + return; + }

          + if (queuedGuaranteedContainers.isEmpty() &&
          + queuedOpportunisticContainers.isEmpty() &&
          + this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + }

          else {
          + LOG.info("No available resources for container {} to start its execution "
          + + "immediately.", container.getContainerId());
          + boolean isQueued = true;
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.GUARANTEED)

          { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + }

          else {
          + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) {
          + LOG.info("Opportunistic container {} will be queued at the NM.",
          + container.getContainerId());
          + queuedOpportunisticContainers.put(
          + container.getContainerId(), container);
          + } else {
          + isQueued = false;
          + LOG.info("Opportunistic container [{}] will not be queued at the NM" +
          + "since max queue length [{}] has been reached",
          + container.getContainerId(), maxOppQueueLength);
          + container.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Opportunistic container queue is full.");
          + }
          + }
          + if (isQueued) {
          + try

          { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + }

          catch (IOException e)

          { + LOG.warn("Could not store container state into store..", e); + }

          + }
          + }
          + }
          +
          + private void killOpportunisticContainers(Container container) {
          + List<Container> extraOpportContainersToKill =
          + pickOpportunisticContainersToKill(container.getContainerId());
          + // Kill the opportunistic containers that were chosen.
          + for (Container contToKill : extraOpportContainersToKill) {
          + contToKill.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Container Killed to make room for Guaranteed Container.");
          + oppContainersMarkedForKill.put(contToKill.getContainerId(), contToKill);
          + LOG.info(
          + "Opportunistic container {} will be killed in order to start the "
          + + "execution of guaranteed container {}.",
          + contToKill.getContainerId(), container.getContainerId());
          + }
          + }
          +
          + private void startAllocatedContainer(Container container) {
          + LOG.info("Starting container [" + container.getContainerId()+ "]");
          + scheduledToRunContainers.put(container.getContainerId(), container);
          + this.utilizationManager.addContainerResources(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + + container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + + container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + + 1); + }

          + container.sendLaunchEvent();
          + }
          +
          + private List<Container> pickOpportunisticContainersToKill(
          + ContainerId containerToStartId) {
          + // The additional opportunistic containers that need to be killed for the
          + // given container to start.
          + List<Container> extraOpportContainersToKill = new ArrayList<>();
          + // Track resources that need to be freed.
          + ResourceUtilization resourcesToFreeUp = resourcesToFreeUp(
          + containerToStartId);
          +
          + // Go over the running opportunistic containers.
          + // Use a descending iterator to kill more recently started containers.
          + Iterator<Container> reverseContainerIterator =
          + new LinkedList<>(scheduledToRunContainers.values()).descendingIterator();
          + while(reverseContainerIterator.hasNext() &&
          + !hasSufficientResources(resourcesToFreeUp)) {
          + Container runningCont = reverseContainerIterator.next();
          + if (runningCont.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC) {
          +
          + if (oppContainersMarkedForKill.containsKey(
          + runningCont.getContainerId()))

          { + // These containers have already been marked to be killed. + // So exclude them.. + continue; + }

          + extraOpportContainersToKill.add(runningCont);
          + ResourceUtilizationManager.decreaseResourceUtilization(
          + getContainersMonitor(), resourcesToFreeUp,
          + runningCont.getResource());
          + }
          + }
          + if (!hasSufficientResources(resourcesToFreeUp)) {
          + LOG.warn("There are no sufficient resources to start guaranteed [{}]" +
          + "even after attempting to kill all running" +
          + "opportunistic containers.", containerToStartId);
          + }
          + return extraOpportContainersToKill;
          + }
          +
          + private boolean hasSufficientResources(
          + ResourceUtilization resourcesToFreeUp)

          { + return resourcesToFreeUp.getPhysicalMemory() <= 0 && + resourcesToFreeUp.getVirtualMemory() <= 0 && + resourcesToFreeUp.getCPU() <= 0.0f; + }

          +
          + private ResourceUtilization resourcesToFreeUp(
          + ContainerId containerToStartId) {
          + // Get allocation of currently allocated containers.
          + ResourceUtilization resourceAllocationToFreeUp = ResourceUtilization
          + .newInstance(this.utilizationManager.getCurrentUtilization());
          +
          + // Add to the allocation the allocation of the pending guaranteed
          + // containers that will start before the current container will be started.
          + for (Container container : queuedGuaranteedContainers.values()) {
          + ResourceUtilizationManager.increaseResourceUtilization(
          + getContainersMonitor(), resourceAllocationToFreeUp,
          + container.getResource());
          + if (container.getContainerId().equals(containerToStartId))

          { + break; + }

          + }
          +
          + // These Resources have already been freed, due to demand from an
          — End diff –

          Rephrase? "These resources are being freed, likely at the behest of another guaranteed container."?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87641557 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedOpportunisticContainers = new LinkedHashMap<>(); + + // Used to keep track of containers that have been marked to be killed + // to make room for a guaranteed container. + private final Map<ContainerId, Container> oppContainersMarkedForKill = + new HashMap<>(); + + // Containers launched by the Scheduler will take a while to actually + // move to the RUNNING state, but should still be fair game for killing + // by the scheduler to make room for guaranteed containers. + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers = + new LinkedHashMap<>(); + + private final ContainerQueuingLimit queuingLimit = + ContainerQueuingLimit.newInstance(); + + private final OpportunisticContainersStatus opportunisticContainersStatus; + + // Resource Utilization Manager that decides how utilization of the cluster + // increase / decreases based on container start / finish + private ResourceUtilizationManager utilizationManager; + + /** + * Instantiate a Container Scheduler. + * @param context NodeManager Context. + */ + public ContainerScheduler(Context context) { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + } + + @VisibleForTesting + public ContainerScheduler(Context context, int qLength) { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + } + + /** + * Handle ContainerSchedulerEvents. + * @param event ContainerSchedulerEvent. + */ + @Override + public void handle(ContainerSchedulerEvent event) { + switch (event.getType()) { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + } + } + + /** + * Return number of queued containers. + * @return Number of queued containers. + */ + public int getNumQueuedContainers() { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedGuaranteedContainers() { + return this.queuedGuaranteedContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedOpportunisticContainers() { + return this.queuedOpportunisticContainers.size(); + } + + public OpportunisticContainersStatus getOpportunisticContainersStatus() { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + } + + private void onContainerCompleted(Container container) { + // decrement only if it was a running container + if (scheduledToRunContainers.containsKey(container.getContainerId())) { + this.utilizationManager.subtractContainerResource(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + } + } + scheduledToRunContainers.remove(container.getContainerId()); + oppContainersMarkedForKill.remove(container.getContainerId()); + startPendingContainers(); + } + + private void startPendingContainers() { + // Start pending guaranteed containers, if resources available. + boolean resourcesAvailable = + startContainersFromQueue(queuedGuaranteedContainers.values()); + // Start opportunistic containers, if resources available. + if (resourcesAvailable) { + startContainersFromQueue(queuedOpportunisticContainers.values()); + } + } + + private boolean startContainersFromQueue( + Collection<Container> queuedContainers) { + Iterator<Container> cIter = queuedContainers.iterator(); + boolean resourcesAvailable = true; + while (cIter.hasNext() && resourcesAvailable) { + Container container = cIter.next(); + if (this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + cIter.remove(); + } else { + resourcesAvailable = false; + } + } + return resourcesAvailable; + } + + @VisibleForTesting + protected void scheduleContainer(Container container) { + if (maxOppQueueLength <= 0) { + startAllocatedContainer(container); + return; + } + if (queuedGuaranteedContainers.isEmpty() && + queuedOpportunisticContainers.isEmpty() && + this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + } else { + LOG.info("No available resources for container {} to start its execution " + + "immediately.", container.getContainerId()); + boolean isQueued = true; + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.GUARANTEED) { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + } else { + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) { + LOG.info("Opportunistic container {} will be queued at the NM.", + container.getContainerId()); + queuedOpportunisticContainers.put( + container.getContainerId(), container); + } else { + isQueued = false; + LOG.info("Opportunistic container [{}] will not be queued at the NM" + + "since max queue length [{}] has been reached", + container.getContainerId(), maxOppQueueLength); + container.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Opportunistic container queue is full."); + } + } + if (isQueued) { + try { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + } catch (IOException e) { + LOG.warn("Could not store container state into store..", e); + } + } + } + } + + private void killOpportunisticContainers(Container container) { + List<Container> extraOpportContainersToKill = + pickOpportunisticContainersToKill(container.getContainerId()); + // Kill the opportunistic containers that were chosen. + for (Container contToKill : extraOpportContainersToKill) { + contToKill.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Container Killed to make room for Guaranteed Container."); + oppContainersMarkedForKill.put(contToKill.getContainerId(), contToKill); + LOG.info( + "Opportunistic container {} will be killed in order to start the " + + "execution of guaranteed container {}.", + contToKill.getContainerId(), container.getContainerId()); + } + } + + private void startAllocatedContainer(Container container) { + LOG.info("Starting container [" + container.getContainerId()+ "] "); + scheduledToRunContainers.put(container.getContainerId(), container); + this.utilizationManager.addContainerResources(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + + container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + + container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + + 1); + } + container.sendLaunchEvent(); + } + + private List<Container> pickOpportunisticContainersToKill( + ContainerId containerToStartId) { + // The additional opportunistic containers that need to be killed for the + // given container to start. + List<Container> extraOpportContainersToKill = new ArrayList<>(); + // Track resources that need to be freed. + ResourceUtilization resourcesToFreeUp = resourcesToFreeUp( + containerToStartId); + + // Go over the running opportunistic containers. + // Use a descending iterator to kill more recently started containers. + Iterator<Container> reverseContainerIterator = + new LinkedList<>(scheduledToRunContainers.values()).descendingIterator(); + while(reverseContainerIterator.hasNext() && + !hasSufficientResources(resourcesToFreeUp)) { + Container runningCont = reverseContainerIterator.next(); + if (runningCont.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + + if (oppContainersMarkedForKill.containsKey( + runningCont.getContainerId())) { + // These containers have already been marked to be killed. + // So exclude them.. + continue; + } + extraOpportContainersToKill.add(runningCont); + ResourceUtilizationManager.decreaseResourceUtilization( + getContainersMonitor(), resourcesToFreeUp, + runningCont.getResource()); + } + } + if (!hasSufficientResources(resourcesToFreeUp)) { + LOG.warn("There are no sufficient resources to start guaranteed [{}] " + + "even after attempting to kill all running" + + "opportunistic containers.", containerToStartId); + } + return extraOpportContainersToKill; + } + + private boolean hasSufficientResources( + ResourceUtilization resourcesToFreeUp) { + return resourcesToFreeUp.getPhysicalMemory() <= 0 && + resourcesToFreeUp.getVirtualMemory() <= 0 && + resourcesToFreeUp.getCPU() <= 0.0f; + } + + private ResourceUtilization resourcesToFreeUp( + ContainerId containerToStartId) { + // Get allocation of currently allocated containers. + ResourceUtilization resourceAllocationToFreeUp = ResourceUtilization + .newInstance(this.utilizationManager.getCurrentUtilization()); + + // Add to the allocation the allocation of the pending guaranteed + // containers that will start before the current container will be started. + for (Container container : queuedGuaranteedContainers.values()) { + ResourceUtilizationManager.increaseResourceUtilization( + getContainersMonitor(), resourceAllocationToFreeUp, + container.getResource()); + if (container.getContainerId().equals(containerToStartId)) { + break; + } + } + + // These Resources have already been freed, due to demand from an — End diff – Rephrase? "These resources are being freed, likely at the behest of another guaranteed container."?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87641844

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          + new HashMap<>();
          +
          + // Containers launched by the Scheduler will take a while to actually
          + // move to the RUNNING state, but should still be fair game for killing
          + // by the scheduler to make room for guaranteed containers.
          + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers =
          + new LinkedHashMap<>();
          +
          + private final ContainerQueuingLimit queuingLimit =
          + ContainerQueuingLimit.newInstance();
          +
          + private final OpportunisticContainersStatus opportunisticContainersStatus;
          +
          + // Resource Utilization Manager that decides how utilization of the cluster
          + // increase / decreases based on container start / finish
          + private ResourceUtilizationManager utilizationManager;
          +
          + /**
          + * Instantiate a Container Scheduler.
          + * @param context NodeManager Context.
          + */
          + public ContainerScheduler(Context context)

          { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + }

          +
          + @VisibleForTesting
          + public ContainerScheduler(Context context, int qLength)

          { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + }

          +
          + /**
          + * Handle ContainerSchedulerEvents.
          + * @param event ContainerSchedulerEvent.
          + */
          + @Override
          + public void handle(ContainerSchedulerEvent event) {
          + switch (event.getType())

          { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + }

          + }
          +
          + /**
          + * Return number of queued containers.
          + * @return Number of queued containers.
          + */
          + public int getNumQueuedContainers()

          { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedGuaranteedContainers()

          { + return this.queuedGuaranteedContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedOpportunisticContainers()

          { + return this.queuedOpportunisticContainers.size(); + }

          +
          + public OpportunisticContainersStatus getOpportunisticContainersStatus()

          { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + }

          +
          + private void onContainerCompleted(Container container) {
          + // decrement only if it was a running container
          + if (scheduledToRunContainers.containsKey(container.getContainerId())) {
          + this.utilizationManager.subtractContainerResource(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + }

          + }
          + scheduledToRunContainers.remove(container.getContainerId());
          + oppContainersMarkedForKill.remove(container.getContainerId());
          + startPendingContainers();
          + }
          +
          + private void startPendingContainers() {
          + // Start pending guaranteed containers, if resources available.
          + boolean resourcesAvailable =
          + startContainersFromQueue(queuedGuaranteedContainers.values());
          + // Start opportunistic containers, if resources available.
          + if (resourcesAvailable)

          { + startContainersFromQueue(queuedOpportunisticContainers.values()); + }

          + }
          +
          + private boolean startContainersFromQueue(
          + Collection<Container> queuedContainers) {
          + Iterator<Container> cIter = queuedContainers.iterator();
          + boolean resourcesAvailable = true;
          + while (cIter.hasNext() && resourcesAvailable) {
          + Container container = cIter.next();
          + if (this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + cIter.remove(); + }

          else

          { + resourcesAvailable = false; + }

          + }
          + return resourcesAvailable;
          + }
          +
          + @VisibleForTesting
          + protected void scheduleContainer(Container container) {
          + if (maxOppQueueLength <= 0)

          { + startAllocatedContainer(container); + return; + }

          + if (queuedGuaranteedContainers.isEmpty() &&
          + queuedOpportunisticContainers.isEmpty() &&
          + this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + }

          else {
          + LOG.info("No available resources for container {} to start its execution "
          + + "immediately.", container.getContainerId());
          + boolean isQueued = true;
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.GUARANTEED)

          { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + }

          else {
          + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) {
          + LOG.info("Opportunistic container {} will be queued at the NM.",
          + container.getContainerId());
          + queuedOpportunisticContainers.put(
          + container.getContainerId(), container);
          + } else {
          + isQueued = false;
          + LOG.info("Opportunistic container [{}] will not be queued at the NM" +
          + "since max queue length [{}] has been reached",
          + container.getContainerId(), maxOppQueueLength);
          + container.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Opportunistic container queue is full.");
          + }
          + }
          + if (isQueued) {
          + try

          { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + }

          catch (IOException e)

          { + LOG.warn("Could not store container state into store..", e); + }

          + }
          + }
          + }
          +
          + private void killOpportunisticContainers(Container container) {
          + List<Container> extraOpportContainersToKill =
          + pickOpportunisticContainersToKill(container.getContainerId());
          + // Kill the opportunistic containers that were chosen.
          + for (Container contToKill : extraOpportContainersToKill) {
          + contToKill.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Container Killed to make room for Guaranteed Container.");
          + oppContainersMarkedForKill.put(contToKill.getContainerId(), contToKill);
          + LOG.info(
          + "Opportunistic container {} will be killed in order to start the "
          + + "execution of guaranteed container {}.",
          + contToKill.getContainerId(), container.getContainerId());
          + }
          + }
          +
          + private void startAllocatedContainer(Container container) {
          + LOG.info("Starting container [" + container.getContainerId()+ "]");
          + scheduledToRunContainers.put(container.getContainerId(), container);
          + this.utilizationManager.addContainerResources(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + + container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + + container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + + 1); + }

          + container.sendLaunchEvent();
          + }
          +
          + private List<Container> pickOpportunisticContainersToKill(
          + ContainerId containerToStartId) {
          + // The additional opportunistic containers that need to be killed for the
          + // given container to start.
          + List<Container> extraOpportContainersToKill = new ArrayList<>();
          + // Track resources that need to be freed.
          + ResourceUtilization resourcesToFreeUp = resourcesToFreeUp(
          + containerToStartId);
          +
          + // Go over the running opportunistic containers.
          + // Use a descending iterator to kill more recently started containers.
          + Iterator<Container> reverseContainerIterator =
          + new LinkedList<>(scheduledToRunContainers.values()).descendingIterator();
          + while(reverseContainerIterator.hasNext() &&
          + !hasSufficientResources(resourcesToFreeUp)) {
          + Container runningCont = reverseContainerIterator.next();
          + if (runningCont.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC) {
          +
          + if (oppContainersMarkedForKill.containsKey(
          + runningCont.getContainerId()))

          { + // These containers have already been marked to be killed. + // So exclude them.. + continue; + }

          + extraOpportContainersToKill.add(runningCont);
          + ResourceUtilizationManager.decreaseResourceUtilization(
          + getContainersMonitor(), resourcesToFreeUp,
          + runningCont.getResource());
          + }
          + }
          + if (!hasSufficientResources(resourcesToFreeUp)) {
          + LOG.warn("There are no sufficient resources to start guaranteed [{}]" +
          + "even after attempting to kill all running" +
          + "opportunistic containers.", containerToStartId);
          + }
          + return extraOpportContainersToKill;
          + }
          +
          + private boolean hasSufficientResources(
          + ResourceUtilization resourcesToFreeUp)

          { + return resourcesToFreeUp.getPhysicalMemory() <= 0 && + resourcesToFreeUp.getVirtualMemory() <= 0 && + resourcesToFreeUp.getCPU() <= 0.0f; + }

          +
          + private ResourceUtilization resourcesToFreeUp(
          + ContainerId containerToStartId) {
          + // Get allocation of currently allocated containers.
          + ResourceUtilization resourceAllocationToFreeUp = ResourceUtilization
          + .newInstance(this.utilizationManager.getCurrentUtilization());
          +
          + // Add to the allocation the allocation of the pending guaranteed
          + // containers that will start before the current container will be started.
          + for (Container container : queuedGuaranteedContainers.values()) {
          + ResourceUtilizationManager.increaseResourceUtilization(
          + getContainersMonitor(), resourceAllocationToFreeUp,
          + container.getResource());
          + if (container.getContainerId().equals(containerToStartId))

          { + break; + }

          + }
          +
          + // These Resources have already been freed, due to demand from an
          + // earlier Guaranteed container.
          + for (Container container : oppContainersMarkedForKill.values())

          { + ResourceUtilizationManager.decreaseResourceUtilization( + getContainersMonitor(), resourceAllocationToFreeUp, + container.getResource()); + }

          +
          + // Subtract the overall node resources.
          + getContainersMonitor().subtractNodeResourcesFromResourceUtilization(
          + resourceAllocationToFreeUp);
          + return resourceAllocationToFreeUp;
          + }
          +
          + public void updateQueuingLimit(ContainerQueuingLimit limit) {
          + this.queuingLimit.setMaxQueueLength(limit.getMaxQueueLength());
          + // TODO: Include wait time as well once it is implemented
          — End diff –

          Annotate the TODO with a JIRA?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87641844 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedOpportunisticContainers = new LinkedHashMap<>(); + + // Used to keep track of containers that have been marked to be killed + // to make room for a guaranteed container. + private final Map<ContainerId, Container> oppContainersMarkedForKill = + new HashMap<>(); + + // Containers launched by the Scheduler will take a while to actually + // move to the RUNNING state, but should still be fair game for killing + // by the scheduler to make room for guaranteed containers. + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers = + new LinkedHashMap<>(); + + private final ContainerQueuingLimit queuingLimit = + ContainerQueuingLimit.newInstance(); + + private final OpportunisticContainersStatus opportunisticContainersStatus; + + // Resource Utilization Manager that decides how utilization of the cluster + // increase / decreases based on container start / finish + private ResourceUtilizationManager utilizationManager; + + /** + * Instantiate a Container Scheduler. + * @param context NodeManager Context. + */ + public ContainerScheduler(Context context) { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + } + + @VisibleForTesting + public ContainerScheduler(Context context, int qLength) { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + } + + /** + * Handle ContainerSchedulerEvents. + * @param event ContainerSchedulerEvent. + */ + @Override + public void handle(ContainerSchedulerEvent event) { + switch (event.getType()) { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + } + } + + /** + * Return number of queued containers. + * @return Number of queued containers. + */ + public int getNumQueuedContainers() { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedGuaranteedContainers() { + return this.queuedGuaranteedContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedOpportunisticContainers() { + return this.queuedOpportunisticContainers.size(); + } + + public OpportunisticContainersStatus getOpportunisticContainersStatus() { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + } + + private void onContainerCompleted(Container container) { + // decrement only if it was a running container + if (scheduledToRunContainers.containsKey(container.getContainerId())) { + this.utilizationManager.subtractContainerResource(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + } + } + scheduledToRunContainers.remove(container.getContainerId()); + oppContainersMarkedForKill.remove(container.getContainerId()); + startPendingContainers(); + } + + private void startPendingContainers() { + // Start pending guaranteed containers, if resources available. + boolean resourcesAvailable = + startContainersFromQueue(queuedGuaranteedContainers.values()); + // Start opportunistic containers, if resources available. + if (resourcesAvailable) { + startContainersFromQueue(queuedOpportunisticContainers.values()); + } + } + + private boolean startContainersFromQueue( + Collection<Container> queuedContainers) { + Iterator<Container> cIter = queuedContainers.iterator(); + boolean resourcesAvailable = true; + while (cIter.hasNext() && resourcesAvailable) { + Container container = cIter.next(); + if (this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + cIter.remove(); + } else { + resourcesAvailable = false; + } + } + return resourcesAvailable; + } + + @VisibleForTesting + protected void scheduleContainer(Container container) { + if (maxOppQueueLength <= 0) { + startAllocatedContainer(container); + return; + } + if (queuedGuaranteedContainers.isEmpty() && + queuedOpportunisticContainers.isEmpty() && + this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + } else { + LOG.info("No available resources for container {} to start its execution " + + "immediately.", container.getContainerId()); + boolean isQueued = true; + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.GUARANTEED) { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + } else { + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) { + LOG.info("Opportunistic container {} will be queued at the NM.", + container.getContainerId()); + queuedOpportunisticContainers.put( + container.getContainerId(), container); + } else { + isQueued = false; + LOG.info("Opportunistic container [{}] will not be queued at the NM" + + "since max queue length [{}] has been reached", + container.getContainerId(), maxOppQueueLength); + container.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Opportunistic container queue is full."); + } + } + if (isQueued) { + try { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + } catch (IOException e) { + LOG.warn("Could not store container state into store..", e); + } + } + } + } + + private void killOpportunisticContainers(Container container) { + List<Container> extraOpportContainersToKill = + pickOpportunisticContainersToKill(container.getContainerId()); + // Kill the opportunistic containers that were chosen. + for (Container contToKill : extraOpportContainersToKill) { + contToKill.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Container Killed to make room for Guaranteed Container."); + oppContainersMarkedForKill.put(contToKill.getContainerId(), contToKill); + LOG.info( + "Opportunistic container {} will be killed in order to start the " + + "execution of guaranteed container {}.", + contToKill.getContainerId(), container.getContainerId()); + } + } + + private void startAllocatedContainer(Container container) { + LOG.info("Starting container [" + container.getContainerId()+ "] "); + scheduledToRunContainers.put(container.getContainerId(), container); + this.utilizationManager.addContainerResources(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + + container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + + container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + + 1); + } + container.sendLaunchEvent(); + } + + private List<Container> pickOpportunisticContainersToKill( + ContainerId containerToStartId) { + // The additional opportunistic containers that need to be killed for the + // given container to start. + List<Container> extraOpportContainersToKill = new ArrayList<>(); + // Track resources that need to be freed. + ResourceUtilization resourcesToFreeUp = resourcesToFreeUp( + containerToStartId); + + // Go over the running opportunistic containers. + // Use a descending iterator to kill more recently started containers. + Iterator<Container> reverseContainerIterator = + new LinkedList<>(scheduledToRunContainers.values()).descendingIterator(); + while(reverseContainerIterator.hasNext() && + !hasSufficientResources(resourcesToFreeUp)) { + Container runningCont = reverseContainerIterator.next(); + if (runningCont.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + + if (oppContainersMarkedForKill.containsKey( + runningCont.getContainerId())) { + // These containers have already been marked to be killed. + // So exclude them.. + continue; + } + extraOpportContainersToKill.add(runningCont); + ResourceUtilizationManager.decreaseResourceUtilization( + getContainersMonitor(), resourcesToFreeUp, + runningCont.getResource()); + } + } + if (!hasSufficientResources(resourcesToFreeUp)) { + LOG.warn("There are no sufficient resources to start guaranteed [{}] " + + "even after attempting to kill all running" + + "opportunistic containers.", containerToStartId); + } + return extraOpportContainersToKill; + } + + private boolean hasSufficientResources( + ResourceUtilization resourcesToFreeUp) { + return resourcesToFreeUp.getPhysicalMemory() <= 0 && + resourcesToFreeUp.getVirtualMemory() <= 0 && + resourcesToFreeUp.getCPU() <= 0.0f; + } + + private ResourceUtilization resourcesToFreeUp( + ContainerId containerToStartId) { + // Get allocation of currently allocated containers. + ResourceUtilization resourceAllocationToFreeUp = ResourceUtilization + .newInstance(this.utilizationManager.getCurrentUtilization()); + + // Add to the allocation the allocation of the pending guaranteed + // containers that will start before the current container will be started. + for (Container container : queuedGuaranteedContainers.values()) { + ResourceUtilizationManager.increaseResourceUtilization( + getContainersMonitor(), resourceAllocationToFreeUp, + container.getResource()); + if (container.getContainerId().equals(containerToStartId)) { + break; + } + } + + // These Resources have already been freed, due to demand from an + // earlier Guaranteed container. + for (Container container : oppContainersMarkedForKill.values()) { + ResourceUtilizationManager.decreaseResourceUtilization( + getContainersMonitor(), resourceAllocationToFreeUp, + container.getResource()); + } + + // Subtract the overall node resources. + getContainersMonitor().subtractNodeResourcesFromResourceUtilization( + resourceAllocationToFreeUp); + return resourceAllocationToFreeUp; + } + + public void updateQueuingLimit(ContainerQueuingLimit limit) { + this.queuingLimit.setMaxQueueLength(limit.getMaxQueueLength()); + // TODO: Include wait time as well once it is implemented — End diff – Annotate the TODO with a JIRA?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87645111

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ResourceUtilizationManager.java —
          @@ -0,0 +1,163 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.Resource;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +/**
          + * This class abstracts out how a container contributes to Resource Utilization.
          + * It is used by the

          {@link ContainerScheduler}

          to determine which
          + * OPPORTUNISTIC containers to be killed to make room for a GUARANTEED
          + * container.
          + * It currently equates resource utilization with the total resource allocated
          + * to the container. Another implementation might choose to use the actual
          + * resource utilization.
          + */
          +
          +public class ResourceUtilizationManager {
          — End diff –

          Should this be called ResourceUtilizationTracker?

          Also, would it make more sense for this to be an interface with an allocation-based implementation? Later, one could add a usage-based implementation and may be integrate tightly with ContainersMonitor?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87645111 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ResourceUtilizationManager.java — @@ -0,0 +1,163 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.Resource; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * This class abstracts out how a container contributes to Resource Utilization. + * It is used by the {@link ContainerScheduler} to determine which + * OPPORTUNISTIC containers to be killed to make room for a GUARANTEED + * container. + * It currently equates resource utilization with the total resource allocated + * to the container. Another implementation might choose to use the actual + * resource utilization. + */ + +public class ResourceUtilizationManager { — End diff – Should this be called ResourceUtilizationTracker? Also, would it make more sense for this to be an interface with an allocation-based implementation? Later, one could add a usage-based implementation and may be integrate tightly with ContainersMonitor?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87643346

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java —
          @@ -311,6 +318,10 @@ protected void createAMRMProxyService(Configuration conf) {
          }
          }

          + protected ContainerScheduler createContainerScheduler(Context cntxt) {
          — End diff –

          Is this method exposed for testing? If yes, should it be marked @VisibleForTesting?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87643346 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java — @@ -311,6 +318,10 @@ protected void createAMRMProxyService(Configuration conf) { } } + protected ContainerScheduler createContainerScheduler(Context cntxt) { — End diff – Is this method exposed for testing? If yes, should it be marked @VisibleForTesting?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87642663

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          + new HashMap<>();
          +
          + // Containers launched by the Scheduler will take a while to actually
          + // move to the RUNNING state, but should still be fair game for killing
          + // by the scheduler to make room for guaranteed containers.
          + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers =
          + new LinkedHashMap<>();
          +
          + private final ContainerQueuingLimit queuingLimit =
          + ContainerQueuingLimit.newInstance();
          +
          + private final OpportunisticContainersStatus opportunisticContainersStatus;
          +
          + // Resource Utilization Manager that decides how utilization of the cluster
          + // increase / decreases based on container start / finish
          + private ResourceUtilizationManager utilizationManager;
          +
          + /**
          + * Instantiate a Container Scheduler.
          + * @param context NodeManager Context.
          + */
          + public ContainerScheduler(Context context)

          { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + }

          +
          + @VisibleForTesting
          + public ContainerScheduler(Context context, int qLength)

          { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + }

          +
          + /**
          + * Handle ContainerSchedulerEvents.
          + * @param event ContainerSchedulerEvent.
          + */
          + @Override
          + public void handle(ContainerSchedulerEvent event) {
          + switch (event.getType())

          { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + }

          + }
          +
          + /**
          + * Return number of queued containers.
          + * @return Number of queued containers.
          + */
          + public int getNumQueuedContainers()

          { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedGuaranteedContainers()

          { + return this.queuedGuaranteedContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedOpportunisticContainers()

          { + return this.queuedOpportunisticContainers.size(); + }

          +
          + public OpportunisticContainersStatus getOpportunisticContainersStatus()

          { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + }

          +
          + private void onContainerCompleted(Container container) {
          + // decrement only if it was a running container
          + if (scheduledToRunContainers.containsKey(container.getContainerId())) {
          + this.utilizationManager.subtractContainerResource(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + }

          + }
          + scheduledToRunContainers.remove(container.getContainerId());
          + oppContainersMarkedForKill.remove(container.getContainerId());
          + startPendingContainers();
          + }
          +
          + private void startPendingContainers() {
          + // Start pending guaranteed containers, if resources available.
          + boolean resourcesAvailable =
          + startContainersFromQueue(queuedGuaranteedContainers.values());
          + // Start opportunistic containers, if resources available.
          + if (resourcesAvailable)

          { + startContainersFromQueue(queuedOpportunisticContainers.values()); + }

          + }
          +
          + private boolean startContainersFromQueue(
          + Collection<Container> queuedContainers) {
          + Iterator<Container> cIter = queuedContainers.iterator();
          + boolean resourcesAvailable = true;
          + while (cIter.hasNext() && resourcesAvailable) {
          + Container container = cIter.next();
          + if (this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + cIter.remove(); + }

          else

          { + resourcesAvailable = false; + }

          + }
          + return resourcesAvailable;
          + }
          +
          + @VisibleForTesting
          + protected void scheduleContainer(Container container) {
          + if (maxOppQueueLength <= 0)

          { + startAllocatedContainer(container); + return; + }

          + if (queuedGuaranteedContainers.isEmpty() &&
          + queuedOpportunisticContainers.isEmpty() &&
          + this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + }

          else {
          + LOG.info("No available resources for container {} to start its execution "
          + + "immediately.", container.getContainerId());
          + boolean isQueued = true;
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.GUARANTEED)

          { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + }

          else {
          + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) {
          + LOG.info("Opportunistic container {} will be queued at the NM.",
          + container.getContainerId());
          + queuedOpportunisticContainers.put(
          + container.getContainerId(), container);
          + } else {
          + isQueued = false;
          + LOG.info("Opportunistic container [{}] will not be queued at the NM" +
          + "since max queue length [{}] has been reached",
          + container.getContainerId(), maxOppQueueLength);
          + container.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Opportunistic container queue is full.");
          + }
          + }
          + if (isQueued) {
          + try

          { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + }

          catch (IOException e)

          { + LOG.warn("Could not store container state into store..", e); + }

          + }
          + }
          + }
          +
          + private void killOpportunisticContainers(Container container) {
          + List<Container> extraOpportContainersToKill =
          + pickOpportunisticContainersToKill(container.getContainerId());
          + // Kill the opportunistic containers that were chosen.
          + for (Container contToKill : extraOpportContainersToKill) {
          + contToKill.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Container Killed to make room for Guaranteed Container.");
          + oppContainersMarkedForKill.put(contToKill.getContainerId(), contToKill);
          + LOG.info(
          + "Opportunistic container {} will be killed in order to start the "
          + + "execution of guaranteed container {}.",
          + contToKill.getContainerId(), container.getContainerId());
          + }
          + }
          +
          + private void startAllocatedContainer(Container container) {
          + LOG.info("Starting container [" + container.getContainerId()+ "]");
          + scheduledToRunContainers.put(container.getContainerId(), container);
          + this.utilizationManager.addContainerResources(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + + container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + + container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + + 1); + }

          + container.sendLaunchEvent();
          + }
          +
          + private List<Container> pickOpportunisticContainersToKill(
          + ContainerId containerToStartId) {
          + // The additional opportunistic containers that need to be killed for the
          + // given container to start.
          + List<Container> extraOpportContainersToKill = new ArrayList<>();
          + // Track resources that need to be freed.
          + ResourceUtilization resourcesToFreeUp = resourcesToFreeUp(
          + containerToStartId);
          +
          + // Go over the running opportunistic containers.
          + // Use a descending iterator to kill more recently started containers.
          + Iterator<Container> reverseContainerIterator =
          + new LinkedList<>(scheduledToRunContainers.values()).descendingIterator();
          + while(reverseContainerIterator.hasNext() &&
          + !hasSufficientResources(resourcesToFreeUp)) {
          + Container runningCont = reverseContainerIterator.next();
          + if (runningCont.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC) {
          +
          + if (oppContainersMarkedForKill.containsKey(
          + runningCont.getContainerId()))

          { + // These containers have already been marked to be killed. + // So exclude them.. + continue; + }

          + extraOpportContainersToKill.add(runningCont);
          + ResourceUtilizationManager.decreaseResourceUtilization(
          + getContainersMonitor(), resourcesToFreeUp,
          + runningCont.getResource());
          + }
          + }
          + if (!hasSufficientResources(resourcesToFreeUp)) {
          + LOG.warn("There are no sufficient resources to start guaranteed [{}]" +
          + "even after attempting to kill all running" +
          + "opportunistic containers.", containerToStartId);
          + }
          + return extraOpportContainersToKill;
          + }
          +
          + private boolean hasSufficientResources(
          + ResourceUtilization resourcesToFreeUp)

          { + return resourcesToFreeUp.getPhysicalMemory() <= 0 && + resourcesToFreeUp.getVirtualMemory() <= 0 && + resourcesToFreeUp.getCPU() <= 0.0f; + }

          +
          + private ResourceUtilization resourcesToFreeUp(
          + ContainerId containerToStartId) {
          + // Get allocation of currently allocated containers.
          + ResourceUtilization resourceAllocationToFreeUp = ResourceUtilization
          + .newInstance(this.utilizationManager.getCurrentUtilization());
          +
          + // Add to the allocation the allocation of the pending guaranteed
          + // containers that will start before the current container will be started.
          + for (Container container : queuedGuaranteedContainers.values()) {
          + ResourceUtilizationManager.increaseResourceUtilization(
          + getContainersMonitor(), resourceAllocationToFreeUp,
          + container.getResource());
          + if (container.getContainerId().equals(containerToStartId))

          { + break; + }

          + }
          +
          + // These Resources have already been freed, due to demand from an
          + // earlier Guaranteed container.
          + for (Container container : oppContainersMarkedForKill.values())

          { + ResourceUtilizationManager.decreaseResourceUtilization( + getContainersMonitor(), resourceAllocationToFreeUp, + container.getResource()); + }

          +
          + // Subtract the overall node resources.
          + getContainersMonitor().subtractNodeResourcesFromResourceUtilization(
          + resourceAllocationToFreeUp);
          + return resourceAllocationToFreeUp;
          + }
          +
          + public void updateQueuingLimit(ContainerQueuingLimit limit) {
          + this.queuingLimit.setMaxQueueLength(limit.getMaxQueueLength());
          + // TODO: Include wait time as well once it is implemented
          + if (this.queuingLimit.getMaxQueueLength() > -1)

          { + shedQueuedOpportunisticContainers(); + }

          + }
          +
          + private void shedQueuedOpportunisticContainers() {
          + int numAllowed = this.queuingLimit.getMaxQueueLength();
          + Iterator<Container> containerIter =
          + queuedOpportunisticContainers.values().iterator();
          + while (containerIter.hasNext()) {
          + Container container = containerIter.next();
          + if (numAllowed <= 0) {
          + container.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Container Killed to make room for Guaranteed Container.");
          — End diff –

          Shouldn't this diagnostic message be similar to the log below?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87642663 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java — @@ -0,0 +1,393 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.yarn.api.records.ContainerExitStatus; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.Context; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * The ContainerScheduler manages a collection of runnable containers. It + * ensures that a container is launched only if all it launch criteria are + * met. It also ensures that OPPORTUNISTIC containers are killed to make + * room for GUARANTEED containers. + */ +public class ContainerScheduler extends AbstractService implements + EventHandler<ContainerSchedulerEvent> { + + private static final Logger LOG = + LoggerFactory.getLogger(ContainerScheduler.class); + + private final Context context; + private final int maxOppQueueLength; + + // Queue of Guaranteed Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedGuaranteedContainers = new LinkedHashMap<>(); + // Queue of Opportunistic Containers waiting for resources to run + private final LinkedHashMap<ContainerId, Container> + queuedOpportunisticContainers = new LinkedHashMap<>(); + + // Used to keep track of containers that have been marked to be killed + // to make room for a guaranteed container. + private final Map<ContainerId, Container> oppContainersMarkedForKill = + new HashMap<>(); + + // Containers launched by the Scheduler will take a while to actually + // move to the RUNNING state, but should still be fair game for killing + // by the scheduler to make room for guaranteed containers. + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers = + new LinkedHashMap<>(); + + private final ContainerQueuingLimit queuingLimit = + ContainerQueuingLimit.newInstance(); + + private final OpportunisticContainersStatus opportunisticContainersStatus; + + // Resource Utilization Manager that decides how utilization of the cluster + // increase / decreases based on container start / finish + private ResourceUtilizationManager utilizationManager; + + /** + * Instantiate a Container Scheduler. + * @param context NodeManager Context. + */ + public ContainerScheduler(Context context) { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + } + + @VisibleForTesting + public ContainerScheduler(Context context, int qLength) { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + } + + /** + * Handle ContainerSchedulerEvents. + * @param event ContainerSchedulerEvent. + */ + @Override + public void handle(ContainerSchedulerEvent event) { + switch (event.getType()) { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + } + } + + /** + * Return number of queued containers. + * @return Number of queued containers. + */ + public int getNumQueuedContainers() { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedGuaranteedContainers() { + return this.queuedGuaranteedContainers.size(); + } + + @VisibleForTesting + public int getNumQueuedOpportunisticContainers() { + return this.queuedOpportunisticContainers.size(); + } + + public OpportunisticContainersStatus getOpportunisticContainersStatus() { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + } + + private void onContainerCompleted(Container container) { + // decrement only if it was a running container + if (scheduledToRunContainers.containsKey(container.getContainerId())) { + this.utilizationManager.subtractContainerResource(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + } + } + scheduledToRunContainers.remove(container.getContainerId()); + oppContainersMarkedForKill.remove(container.getContainerId()); + startPendingContainers(); + } + + private void startPendingContainers() { + // Start pending guaranteed containers, if resources available. + boolean resourcesAvailable = + startContainersFromQueue(queuedGuaranteedContainers.values()); + // Start opportunistic containers, if resources available. + if (resourcesAvailable) { + startContainersFromQueue(queuedOpportunisticContainers.values()); + } + } + + private boolean startContainersFromQueue( + Collection<Container> queuedContainers) { + Iterator<Container> cIter = queuedContainers.iterator(); + boolean resourcesAvailable = true; + while (cIter.hasNext() && resourcesAvailable) { + Container container = cIter.next(); + if (this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + cIter.remove(); + } else { + resourcesAvailable = false; + } + } + return resourcesAvailable; + } + + @VisibleForTesting + protected void scheduleContainer(Container container) { + if (maxOppQueueLength <= 0) { + startAllocatedContainer(container); + return; + } + if (queuedGuaranteedContainers.isEmpty() && + queuedOpportunisticContainers.isEmpty() && + this.utilizationManager.hasResourcesAvailable(container)) { + startAllocatedContainer(container); + } else { + LOG.info("No available resources for container {} to start its execution " + + "immediately.", container.getContainerId()); + boolean isQueued = true; + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.GUARANTEED) { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + } else { + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) { + LOG.info("Opportunistic container {} will be queued at the NM.", + container.getContainerId()); + queuedOpportunisticContainers.put( + container.getContainerId(), container); + } else { + isQueued = false; + LOG.info("Opportunistic container [{}] will not be queued at the NM" + + "since max queue length [{}] has been reached", + container.getContainerId(), maxOppQueueLength); + container.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Opportunistic container queue is full."); + } + } + if (isQueued) { + try { + this.context.getNMStateStore().storeContainerQueued( + container.getContainerId()); + } catch (IOException e) { + LOG.warn("Could not store container state into store..", e); + } + } + } + } + + private void killOpportunisticContainers(Container container) { + List<Container> extraOpportContainersToKill = + pickOpportunisticContainersToKill(container.getContainerId()); + // Kill the opportunistic containers that were chosen. + for (Container contToKill : extraOpportContainersToKill) { + contToKill.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Container Killed to make room for Guaranteed Container."); + oppContainersMarkedForKill.put(contToKill.getContainerId(), contToKill); + LOG.info( + "Opportunistic container {} will be killed in order to start the " + + "execution of guaranteed container {}.", + contToKill.getContainerId(), container.getContainerId()); + } + } + + private void startAllocatedContainer(Container container) { + LOG.info("Starting container [" + container.getContainerId()+ "] "); + scheduledToRunContainers.put(container.getContainerId(), container); + this.utilizationManager.addContainerResources(container); + if (container.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + + container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + + container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + + 1); + } + container.sendLaunchEvent(); + } + + private List<Container> pickOpportunisticContainersToKill( + ContainerId containerToStartId) { + // The additional opportunistic containers that need to be killed for the + // given container to start. + List<Container> extraOpportContainersToKill = new ArrayList<>(); + // Track resources that need to be freed. + ResourceUtilization resourcesToFreeUp = resourcesToFreeUp( + containerToStartId); + + // Go over the running opportunistic containers. + // Use a descending iterator to kill more recently started containers. + Iterator<Container> reverseContainerIterator = + new LinkedList<>(scheduledToRunContainers.values()).descendingIterator(); + while(reverseContainerIterator.hasNext() && + !hasSufficientResources(resourcesToFreeUp)) { + Container runningCont = reverseContainerIterator.next(); + if (runningCont.getContainerTokenIdentifier().getExecutionType() == + ExecutionType.OPPORTUNISTIC) { + + if (oppContainersMarkedForKill.containsKey( + runningCont.getContainerId())) { + // These containers have already been marked to be killed. + // So exclude them.. + continue; + } + extraOpportContainersToKill.add(runningCont); + ResourceUtilizationManager.decreaseResourceUtilization( + getContainersMonitor(), resourcesToFreeUp, + runningCont.getResource()); + } + } + if (!hasSufficientResources(resourcesToFreeUp)) { + LOG.warn("There are no sufficient resources to start guaranteed [{}] " + + "even after attempting to kill all running" + + "opportunistic containers.", containerToStartId); + } + return extraOpportContainersToKill; + } + + private boolean hasSufficientResources( + ResourceUtilization resourcesToFreeUp) { + return resourcesToFreeUp.getPhysicalMemory() <= 0 && + resourcesToFreeUp.getVirtualMemory() <= 0 && + resourcesToFreeUp.getCPU() <= 0.0f; + } + + private ResourceUtilization resourcesToFreeUp( + ContainerId containerToStartId) { + // Get allocation of currently allocated containers. + ResourceUtilization resourceAllocationToFreeUp = ResourceUtilization + .newInstance(this.utilizationManager.getCurrentUtilization()); + + // Add to the allocation the allocation of the pending guaranteed + // containers that will start before the current container will be started. + for (Container container : queuedGuaranteedContainers.values()) { + ResourceUtilizationManager.increaseResourceUtilization( + getContainersMonitor(), resourceAllocationToFreeUp, + container.getResource()); + if (container.getContainerId().equals(containerToStartId)) { + break; + } + } + + // These Resources have already been freed, due to demand from an + // earlier Guaranteed container. + for (Container container : oppContainersMarkedForKill.values()) { + ResourceUtilizationManager.decreaseResourceUtilization( + getContainersMonitor(), resourceAllocationToFreeUp, + container.getResource()); + } + + // Subtract the overall node resources. + getContainersMonitor().subtractNodeResourcesFromResourceUtilization( + resourceAllocationToFreeUp); + return resourceAllocationToFreeUp; + } + + public void updateQueuingLimit(ContainerQueuingLimit limit) { + this.queuingLimit.setMaxQueueLength(limit.getMaxQueueLength()); + // TODO: Include wait time as well once it is implemented + if (this.queuingLimit.getMaxQueueLength() > -1) { + shedQueuedOpportunisticContainers(); + } + } + + private void shedQueuedOpportunisticContainers() { + int numAllowed = this.queuingLimit.getMaxQueueLength(); + Iterator<Container> containerIter = + queuedOpportunisticContainers.values().iterator(); + while (containerIter.hasNext()) { + Container container = containerIter.next(); + if (numAllowed <= 0) { + container.sendKillEvent( + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER, + "Container Killed to make room for Guaranteed Container."); — End diff – Shouldn't this diagnostic message be similar to the log below?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87643223

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManager.java —
          @@ -26,6 +26,8 @@
          import org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEvent;
          import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor
          .ContainersMonitor;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler
          + .ContainerScheduler;
          — End diff –

          imports should be on a single line?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87643223 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManager.java — @@ -26,6 +26,8 @@ import org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEvent; import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor .ContainersMonitor; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler + .ContainerScheduler; — End diff – imports should be on a single line?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87642982

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerSchedulerEvent.java —
          @@ -0,0 +1,51 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import org.apache.hadoop.yarn.event.AbstractEvent;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container
          + .Container;
          — End diff –

          One line for imports?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87642982 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerSchedulerEvent.java — @@ -0,0 +1,51 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import org.apache.hadoop.yarn.event.AbstractEvent; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container + .Container; — End diff – One line for imports?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87645581

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ResourceUtilizationManager.java —
          @@ -0,0 +1,163 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.Resource;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +/**
          + * This class abstracts out how a container contributes to Resource Utilization.
          + * It is used by the

          {@link ContainerScheduler}

          to determine which
          + * OPPORTUNISTIC containers to be killed to make room for a GUARANTEED
          + * container.
          + * It currently equates resource utilization with the total resource allocated
          + * to the container. Another implementation might choose to use the actual
          + * resource utilization.
          + */
          +
          +public class ResourceUtilizationManager {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ResourceUtilizationManager.class);
          +
          + private ResourceUtilization containersAllocation;
          + private ContainerScheduler scheduler;
          +
          + ResourceUtilizationManager(ContainerScheduler scheduler)

          { + this.containersAllocation = ResourceUtilization.newInstance(0, 0, 0.0f); + this.scheduler = scheduler; + }

          +
          + /**
          + * Get the current accumulated utilization. Currently it is the accumulation
          + * of totally allocated resources to a container.
          + * @return ResourceUtilization Resource Utilization.
          + */
          + public ResourceUtilization getCurrentUtilization()

          { + return this.containersAllocation; + }

          +
          + /**
          + * Add Container's resources to the accumulated Utilization.
          + * @param container Container.
          + */
          + public void addContainerResources(Container container)

          { + increaseResourceUtilization( + getContainersMonitor(), this.containersAllocation, + container.getResource()); + }

          +
          + /**
          + * Subtract Container's resources to the accumulated Utilization.
          + * @param container Container.
          + */
          + public void subtractContainerResource(Container container)

          { + decreaseResourceUtilization( + getContainersMonitor(), this.containersAllocation, + container.getResource()); + }

          +
          + /**
          + * Check if NM has resources available currently to run the container.
          + * @param container Container.
          + * @return True, if NM has resources available currently to run the container.
          + */
          + public boolean hasResourcesAvailable(Container container)

          { + long pMemBytes = container.getResource().getMemorySize() * 1024 * 1024L; + return hasResourcesAvailable(pMemBytes, + (long) (getContainersMonitor().getVmemRatio()* pMemBytes), + container.getResource().getVirtualCores()); + }

          +
          + private boolean hasResourcesAvailable(long pMemBytes, long vMemBytes,
          + int cpuVcores) {
          + // Check physical memory.
          + if (LOG.isDebugEnabled()) {
          + LOG.debug("pMemCheck [current={} + asked={} > allowed={}]",
          + this.containersAllocation.getPhysicalMemory(),
          + (pMemBytes >> 20),
          + (getContainersMonitor().getPmemAllocatedForContainers() >> 20));
          + }
          + if (this.containersAllocation.getPhysicalMemory() +
          + (int) (pMemBytes >> 20) >
          + (int) (getContainersMonitor()
          + .getPmemAllocatedForContainers() >> 20))

          { + return false; + }
          +
          + if (LOG.isDebugEnabled()) {
          + LOG.debug("before vMemCheck" +
          + "[isEnabled={}, current={} + asked={} > allowed={}]",
          + getContainersMonitor().isVmemCheckEnabled(),
          + this.containersAllocation.getVirtualMemory(), (vMemBytes >> 20),
          + (getContainersMonitor().getVmemAllocatedForContainers() >> 20));
          + }
          + // Check virtual memory.
          + if (getContainersMonitor().isVmemCheckEnabled() &&
          + this.containersAllocation.getVirtualMemory() +
          + (int) (vMemBytes >> 20) >
          + (int) (getContainersMonitor()
          + .getVmemAllocatedForContainers() >> 20)) { + return false; + }

          +
          + float vCores = (float) cpuVcores /
          + getContainersMonitor().getVCoresAllocatedForContainers();
          + if (LOG.isDebugEnabled()) {
          + LOG.debug("before cpuCheck [asked={} > allowed={}]",
          + this.containersAllocation.getCPU(), vCores);
          + }
          + // Check CPU.
          + if (this.containersAllocation.getCPU() + vCores > 1.0f)

          { + return false; + }

          + return true;
          + }
          +
          + public ContainersMonitor getContainersMonitor()

          { + return this.scheduler.getContainersMonitor(); + }

          +
          + public static void increaseResourceUtilization(
          + ContainersMonitor containersMonitor, ResourceUtilization resourceAlloc,
          + Resource resource)

          { + float vCores = (float) resource.getVirtualCores() / + containersMonitor.getVCoresAllocatedForContainers(); + int vmem = (int) (resource.getMemorySize() + * containersMonitor.getVmemRatio()); + resourceAlloc.addTo((int)resource.getMemorySize(), vmem, vCores); + }

          +
          + public static void decreaseResourceUtilization(
          — End diff –

          Shouldn't this be a non-static method on ResourceUtilization instead?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87645581 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ResourceUtilizationManager.java — @@ -0,0 +1,163 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler; + +import org.apache.hadoop.yarn.api.records.ExecutionType; +import org.apache.hadoop.yarn.api.records.Resource; +import org.apache.hadoop.yarn.api.records.ResourceUtilization; +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * This class abstracts out how a container contributes to Resource Utilization. + * It is used by the {@link ContainerScheduler} to determine which + * OPPORTUNISTIC containers to be killed to make room for a GUARANTEED + * container. + * It currently equates resource utilization with the total resource allocated + * to the container. Another implementation might choose to use the actual + * resource utilization. + */ + +public class ResourceUtilizationManager { + + private static final Logger LOG = + LoggerFactory.getLogger(ResourceUtilizationManager.class); + + private ResourceUtilization containersAllocation; + private ContainerScheduler scheduler; + + ResourceUtilizationManager(ContainerScheduler scheduler) { + this.containersAllocation = ResourceUtilization.newInstance(0, 0, 0.0f); + this.scheduler = scheduler; + } + + /** + * Get the current accumulated utilization. Currently it is the accumulation + * of totally allocated resources to a container. + * @return ResourceUtilization Resource Utilization. + */ + public ResourceUtilization getCurrentUtilization() { + return this.containersAllocation; + } + + /** + * Add Container's resources to the accumulated Utilization. + * @param container Container. + */ + public void addContainerResources(Container container) { + increaseResourceUtilization( + getContainersMonitor(), this.containersAllocation, + container.getResource()); + } + + /** + * Subtract Container's resources to the accumulated Utilization. + * @param container Container. + */ + public void subtractContainerResource(Container container) { + decreaseResourceUtilization( + getContainersMonitor(), this.containersAllocation, + container.getResource()); + } + + /** + * Check if NM has resources available currently to run the container. + * @param container Container. + * @return True, if NM has resources available currently to run the container. + */ + public boolean hasResourcesAvailable(Container container) { + long pMemBytes = container.getResource().getMemorySize() * 1024 * 1024L; + return hasResourcesAvailable(pMemBytes, + (long) (getContainersMonitor().getVmemRatio()* pMemBytes), + container.getResource().getVirtualCores()); + } + + private boolean hasResourcesAvailable(long pMemBytes, long vMemBytes, + int cpuVcores) { + // Check physical memory. + if (LOG.isDebugEnabled()) { + LOG.debug("pMemCheck [current={} + asked={} > allowed={}] ", + this.containersAllocation.getPhysicalMemory(), + (pMemBytes >> 20), + (getContainersMonitor().getPmemAllocatedForContainers() >> 20)); + } + if (this.containersAllocation.getPhysicalMemory() + + (int) (pMemBytes >> 20) > + (int) (getContainersMonitor() + .getPmemAllocatedForContainers() >> 20)) { + return false; + } + + if (LOG.isDebugEnabled()) { + LOG.debug("before vMemCheck" + + " [isEnabled={}, current={} + asked={} > allowed={}] ", + getContainersMonitor().isVmemCheckEnabled(), + this.containersAllocation.getVirtualMemory(), (vMemBytes >> 20), + (getContainersMonitor().getVmemAllocatedForContainers() >> 20)); + } + // Check virtual memory. + if (getContainersMonitor().isVmemCheckEnabled() && + this.containersAllocation.getVirtualMemory() + + (int) (vMemBytes >> 20) > + (int) (getContainersMonitor() + .getVmemAllocatedForContainers() >> 20)) { + return false; + } + + float vCores = (float) cpuVcores / + getContainersMonitor().getVCoresAllocatedForContainers(); + if (LOG.isDebugEnabled()) { + LOG.debug("before cpuCheck [asked={} > allowed={}] ", + this.containersAllocation.getCPU(), vCores); + } + // Check CPU. + if (this.containersAllocation.getCPU() + vCores > 1.0f) { + return false; + } + return true; + } + + public ContainersMonitor getContainersMonitor() { + return this.scheduler.getContainersMonitor(); + } + + public static void increaseResourceUtilization( + ContainersMonitor containersMonitor, ResourceUtilization resourceAlloc, + Resource resource) { + float vCores = (float) resource.getVirtualCores() / + containersMonitor.getVCoresAllocatedForContainers(); + int vmem = (int) (resource.getMemorySize() + * containersMonitor.getVmemRatio()); + resourceAlloc.addTo((int)resource.getMemorySize(), vmem, vCores); + } + + public static void decreaseResourceUtilization( — End diff – Shouldn't this be a non-static method on ResourceUtilization instead?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87644035

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java —
          @@ -743,6 +739,8 @@ private void changeContainerResource(
          LOG.warn("Container " + containerId.toString() + "does not exist");
          return;
          }
          + // TODO: Route this through the ContainerScheduler to
          — End diff –

          Annotate TODO with JIRA?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/143#discussion_r87644035 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java — @@ -743,6 +739,8 @@ private void changeContainerResource( LOG.warn("Container " + containerId.toString() + "does not exist"); return; } + // TODO: Route this through the ContainerScheduler to — End diff – Annotate TODO with JIRA?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/143#discussion_r87641199

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java —
          @@ -0,0 +1,393 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
          +
          +import com.google.common.annotations.VisibleForTesting;
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
          +import org.apache.hadoop.yarn.api.records.ContainerId;
          +import org.apache.hadoop.yarn.api.records.ExecutionType;
          +import org.apache.hadoop.yarn.api.records.ResourceUtilization;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit;
          +import org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus;
          +import org.apache.hadoop.yarn.server.nodemanager.Context;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
          +import org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitor;
          +
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.IOException;
          +import java.util.ArrayList;
          +import java.util.Collection;
          +import java.util.HashMap;
          +import java.util.Iterator;
          +import java.util.LinkedHashMap;
          +import java.util.LinkedList;
          +import java.util.List;
          +import java.util.Map;
          +
          +/**
          + * The ContainerScheduler manages a collection of runnable containers. It
          + * ensures that a container is launched only if all it launch criteria are
          + * met. It also ensures that OPPORTUNISTIC containers are killed to make
          + * room for GUARANTEED containers.
          + */
          +public class ContainerScheduler extends AbstractService implements
          + EventHandler<ContainerSchedulerEvent> {
          +
          + private static final Logger LOG =
          + LoggerFactory.getLogger(ContainerScheduler.class);
          +
          + private final Context context;
          + private final int maxOppQueueLength;
          +
          + // Queue of Guaranteed Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedGuaranteedContainers = new LinkedHashMap<>();
          + // Queue of Opportunistic Containers waiting for resources to run
          + private final LinkedHashMap<ContainerId, Container>
          + queuedOpportunisticContainers = new LinkedHashMap<>();
          +
          + // Used to keep track of containers that have been marked to be killed
          + // to make room for a guaranteed container.
          + private final Map<ContainerId, Container> oppContainersMarkedForKill =
          + new HashMap<>();
          +
          + // Containers launched by the Scheduler will take a while to actually
          + // move to the RUNNING state, but should still be fair game for killing
          + // by the scheduler to make room for guaranteed containers.
          + private final LinkedHashMap<ContainerId, Container> scheduledToRunContainers =
          + new LinkedHashMap<>();
          +
          + private final ContainerQueuingLimit queuingLimit =
          + ContainerQueuingLimit.newInstance();
          +
          + private final OpportunisticContainersStatus opportunisticContainersStatus;
          +
          + // Resource Utilization Manager that decides how utilization of the cluster
          + // increase / decreases based on container start / finish
          + private ResourceUtilizationManager utilizationManager;
          +
          + /**
          + * Instantiate a Container Scheduler.
          + * @param context NodeManager Context.
          + */
          + public ContainerScheduler(Context context)

          { + this(context, context.getConf().getInt( + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH, + YarnConfiguration.NM_OPPORTUNISTIC_CONTAINERS_MAX_QUEUE_LENGTH_DEFAULT)); + }

          +
          + @VisibleForTesting
          + public ContainerScheduler(Context context, int qLength)

          { + super(ContainerScheduler.class.getName()); + this.context = context; + this.maxOppQueueLength = (qLength <= 0) ? 0 : qLength; + this.utilizationManager = new ResourceUtilizationManager(this); + this.opportunisticContainersStatus = + OpportunisticContainersStatus.newInstance(); + }

          +
          + /**
          + * Handle ContainerSchedulerEvents.
          + * @param event ContainerSchedulerEvent.
          + */
          + @Override
          + public void handle(ContainerSchedulerEvent event) {
          + switch (event.getType())

          { + case SCHEDULE_CONTAINER: + scheduleContainer(event.getContainer()); + break; + case CONTAINER_COMPLETED: + onContainerCompleted(event.getContainer()); + break; + default: + LOG.error("Unknown event arrived at ContainerScheduler: " + + event.toString()); + }

          + }
          +
          + /**
          + * Return number of queued containers.
          + * @return Number of queued containers.
          + */
          + public int getNumQueuedContainers()

          { + return this.queuedGuaranteedContainers.size() + + this.queuedOpportunisticContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedGuaranteedContainers()

          { + return this.queuedGuaranteedContainers.size(); + }

          +
          + @VisibleForTesting
          + public int getNumQueuedOpportunisticContainers()

          { + return this.queuedOpportunisticContainers.size(); + }

          +
          + public OpportunisticContainersStatus getOpportunisticContainersStatus()

          { + this.opportunisticContainersStatus.setQueuedOpportContainers( + getNumQueuedOpportunisticContainers()); + this.opportunisticContainersStatus.setWaitQueueLength( + getNumQueuedContainers()); + return this.opportunisticContainersStatus; + }

          +
          + private void onContainerCompleted(Container container) {
          + // decrement only if it was a running container
          + if (scheduledToRunContainers.containsKey(container.getContainerId())) {
          + this.utilizationManager.subtractContainerResource(container);
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.OPPORTUNISTIC)

          { + this.opportunisticContainersStatus.setOpportMemoryUsed( + this.opportunisticContainersStatus.getOpportMemoryUsed() + - container.getResource().getMemorySize()); + this.opportunisticContainersStatus.setOpportCoresUsed( + this.opportunisticContainersStatus.getOpportCoresUsed() + - container.getResource().getVirtualCores()); + this.opportunisticContainersStatus.setRunningOpportContainers( + this.opportunisticContainersStatus.getRunningOpportContainers() + - 1); + }

          + }
          + scheduledToRunContainers.remove(container.getContainerId());
          + oppContainersMarkedForKill.remove(container.getContainerId());
          + startPendingContainers();
          + }
          +
          + private void startPendingContainers() {
          + // Start pending guaranteed containers, if resources available.
          + boolean resourcesAvailable =
          + startContainersFromQueue(queuedGuaranteedContainers.values());
          + // Start opportunistic containers, if resources available.
          + if (resourcesAvailable)

          { + startContainersFromQueue(queuedOpportunisticContainers.values()); + }

          + }
          +
          + private boolean startContainersFromQueue(
          + Collection<Container> queuedContainers) {
          + Iterator<Container> cIter = queuedContainers.iterator();
          + boolean resourcesAvailable = true;
          + while (cIter.hasNext() && resourcesAvailable) {
          + Container container = cIter.next();
          + if (this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + cIter.remove(); + }

          else

          { + resourcesAvailable = false; + }

          + }
          + return resourcesAvailable;
          + }
          +
          + @VisibleForTesting
          + protected void scheduleContainer(Container container) {
          + if (maxOppQueueLength <= 0)

          { + startAllocatedContainer(container); + return; + }

          + if (queuedGuaranteedContainers.isEmpty() &&
          + queuedOpportunisticContainers.isEmpty() &&
          + this.utilizationManager.hasResourcesAvailable(container))

          { + startAllocatedContainer(container); + }

          else {
          + LOG.info("No available resources for container {} to start its execution "
          + + "immediately.", container.getContainerId());
          + boolean isQueued = true;
          + if (container.getContainerTokenIdentifier().getExecutionType() ==
          + ExecutionType.GUARANTEED)

          { + queuedGuaranteedContainers.put(container.getContainerId(), container); + // Kill running opportunistic containers to make space for + // guaranteed container. + killOpportunisticContainers(container); + }

          else {
          + if (queuedOpportunisticContainers.size() <= maxOppQueueLength) {
          + LOG.info("Opportunistic container {} will be queued at the NM.",
          + container.getContainerId());
          + queuedOpportunisticContainers.put(
          + container.getContainerId(), container);
          + } else {
          + isQueued = false;
          + LOG.info("Opportunistic container [{}] will not be queued at the NM" +
          + "since max queue length [{}] has been reached",
          + container.getContainerId(), maxOppQueueLength);
          + container.sendKillEvent(
          + ContainerExitStatus.KILLED_BY_CONTAINER_SCHEDULER,
          + "Opportunistic container queue is full.");
          + }
          + }
          + if (isQueued) {
          + try

          { + this.context.getNMStateStore().stor