Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5220 Scheduling of OPPORTUNISTIC containers through YARN RM
  3. YARN-5486

Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: resourcemanager
    • Labels:
      None

      Description

      YARN-5457 refactors the Distributed Scheduling framework to move the container allocator to yarn-server-common.

      This JIRA proposes to update the allocate method in the new AM service to use the OpportunisticContainerAllocator to allocate opportunistic containers.

      1. YARN-5486.001.patch
        50 kB
        Arun Suresh
      2. YARN-5486.002.patch
        64 kB
        Konstantinos Karanasos
      3. YARN-5486.003.patch
        63 kB
        Konstantinos Karanasos
      4. YARN-5486.004.patch
        63 kB
        Konstantinos Karanasos

        Issue Links

          Activity

          Hide
          asuresh Arun Suresh added a comment -

          Attaching initial patch based on internal prototype by Konstantinos Karanasos and integrating with YARN-5457.

          feedback welcome..

          Show
          asuresh Arun Suresh added a comment - Attaching initial patch based on internal prototype by Konstantinos Karanasos and integrating with YARN-5457 . feedback welcome..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 6m 40s trunk passed
          +1 compile 2m 15s trunk passed
          +1 checkstyle 0m 37s trunk passed
          +1 mvnsite 1m 51s trunk passed
          +1 mvneclipse 0m 57s trunk passed
          +1 findbugs 2m 41s trunk passed
          +1 javadoc 1m 6s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 28s the patch passed
          +1 compile 2m 14s the patch passed
          -1 javac 2m 14s hadoop-yarn-project_hadoop-yarn generated 2 new + 35 unchanged - 0 fixed = 37 total (was 35)
          -1 checkstyle 0m 36s hadoop-yarn-project/hadoop-yarn: The patch generated 30 new + 86 unchanged - 0 fixed = 116 total (was 86)
          +1 mvnsite 1m 46s the patch passed
          +1 mvneclipse 0m 49s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 4s the patch passed
          +1 javadoc 0m 57s the patch passed
          +1 unit 0m 25s hadoop-yarn-server-common in the patch passed.
          +1 unit 13m 14s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 33m 48s hadoop-yarn-server-resourcemanager in the patch failed.
          -1 unit 16m 28s hadoop-yarn-client in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          92m 52s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
            hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService
            hadoop.yarn.client.api.impl.TestYarnClient



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825477/YARN-5486.001.patch
          JIRA Issue YARN-5486
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 92bd012dc360 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 52b25fe
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/12895/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12895/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12895/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12895/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12895/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/12895/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12895/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12895/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 40s trunk passed +1 compile 2m 15s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 1m 51s trunk passed +1 mvneclipse 0m 57s trunk passed +1 findbugs 2m 41s trunk passed +1 javadoc 1m 6s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 2m 14s the patch passed -1 javac 2m 14s hadoop-yarn-project_hadoop-yarn generated 2 new + 35 unchanged - 0 fixed = 37 total (was 35) -1 checkstyle 0m 36s hadoop-yarn-project/hadoop-yarn: The patch generated 30 new + 86 unchanged - 0 fixed = 116 total (was 86) +1 mvnsite 1m 46s the patch passed +1 mvneclipse 0m 49s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 4s the patch passed +1 javadoc 0m 57s the patch passed +1 unit 0m 25s hadoop-yarn-server-common in the patch passed. +1 unit 13m 14s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 33m 48s hadoop-yarn-server-resourcemanager in the patch failed. -1 unit 16m 28s hadoop-yarn-client in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 92m 52s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService   hadoop.yarn.client.api.impl.TestYarnClient Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825477/YARN-5486.001.patch JIRA Issue YARN-5486 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 92bd012dc360 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 52b25fe Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/12895/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12895/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12895/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12895/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12895/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/12895/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12895/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12895/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Arun Suresh for the patch. I looked at it and have a few questions:

          • Why do we need LinkedHashMap in OpportunisticContainerContext?
          • Why are reverting OPPORTUNISTIC_CONTAINER_ALLOCATION_ENABLED to DIST_SCHEDULING_ENABLED?
          • TestOpportunisticsContainerAllocation seems to have redundant harness code from TestAMRMClient?
          • I don't see assertions on containerType in the allocation part of TestOpportunisticsContainerAllocation?
          • Can we also fix the sleep logic in TestOpportunisticsContainerAllocation (Refer to discussion on YARN-5389/YARN-5393)?
          Show
          subru Subru Krishnan added a comment - Thanks Arun Suresh for the patch. I looked at it and have a few questions: Why do we need LinkedHashMap in OpportunisticContainerContext ? Why are reverting OPPORTUNISTIC_CONTAINER_ALLOCATION_ENABLED to DIST_SCHEDULING_ENABLED? TestOpportunisticsContainerAllocation seems to have redundant harness code from TestAMRMClient ? I don't see assertions on containerType in the allocation part of TestOpportunisticsContainerAllocation ? Can we also fix the sleep logic in TestOpportunisticsContainerAllocation (Refer to discussion on YARN-5389 / YARN-5393 )?
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Rebasing against trunk and adding new patch, also including some more changes/fixes.

          Thanks for the feedback, Subru Krishnan.
          Regarding the LinkedHashMap in the OpportunisticContainerContext, we actually need it to keep the ordering of the nodes. Least loaded nodes should come first when iterating over the hashmap, as they should be preferred when placing opportunistic containers.
          I added checks for the ContainerTypes in TestOpportunisticContainersAllocation, as you suggested.
          I suggest to keep the existing sleep logic for now, if it's OK, since it does not seem to make the code much cleaner in the particular cases it is used (also chatted with Chris Douglas about this).

          Show
          kkaranasos Konstantinos Karanasos added a comment - Rebasing against trunk and adding new patch, also including some more changes/fixes. Thanks for the feedback, Subru Krishnan . Regarding the LinkedHashMap in the OpportunisticContainerContext , we actually need it to keep the ordering of the nodes. Least loaded nodes should come first when iterating over the hashmap, as they should be preferred when placing opportunistic containers. I added checks for the ContainerTypes in TestOpportunisticContainersAllocation , as you suggested. I suggest to keep the existing sleep logic for now, if it's OK, since it does not seem to make the code much cleaner in the particular cases it is used (also chatted with Chris Douglas about this).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 13s Maven dependency ordering for branch
          +1 mvninstall 8m 16s trunk passed
          +1 compile 2m 43s trunk passed
          +1 checkstyle 0m 40s trunk passed
          +1 mvnsite 2m 5s trunk passed
          +1 mvneclipse 1m 6s trunk passed
          +1 findbugs 3m 2s trunk passed
          +1 javadoc 1m 10s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 49s the patch passed
          +1 compile 2m 48s the patch passed
          -1 javac 2m 48s hadoop-yarn-project_hadoop-yarn generated 2 new + 35 unchanged - 0 fixed = 37 total (was 35)
          -1 checkstyle 0m 43s hadoop-yarn-project/hadoop-yarn: The patch generated 20 new + 140 unchanged - 0 fixed = 160 total (was 140)
          +1 mvnsite 2m 1s the patch passed
          +1 mvneclipse 0m 54s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 41s the patch passed
          +1 javadoc 1m 6s the patch passed
          +1 unit 0m 30s hadoop-yarn-server-common in the patch passed.
          -1 unit 14m 56s hadoop-yarn-server-nodemanager in the patch failed.
          -1 unit 35m 44s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 unit 16m 36s hadoop-yarn-client in the patch passed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          101m 58s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler
            hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830575/YARN-5486.002.patch
          JIRA Issue YARN-5486
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a32d2a14cf8f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1831be8
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/13229/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13229/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13229/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13229/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13229/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/13229/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13229/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13229/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 8m 16s trunk passed +1 compile 2m 43s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 2m 5s trunk passed +1 mvneclipse 1m 6s trunk passed +1 findbugs 3m 2s trunk passed +1 javadoc 1m 10s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 49s the patch passed +1 compile 2m 48s the patch passed -1 javac 2m 48s hadoop-yarn-project_hadoop-yarn generated 2 new + 35 unchanged - 0 fixed = 37 total (was 35) -1 checkstyle 0m 43s hadoop-yarn-project/hadoop-yarn: The patch generated 20 new + 140 unchanged - 0 fixed = 160 total (was 140) +1 mvnsite 2m 1s the patch passed +1 mvneclipse 0m 54s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 41s the patch passed +1 javadoc 1m 6s the patch passed +1 unit 0m 30s hadoop-yarn-server-common in the patch passed. -1 unit 14m 56s hadoop-yarn-server-nodemanager in the patch failed. -1 unit 35m 44s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 16m 36s hadoop-yarn-client in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 101m 58s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler   hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830575/YARN-5486.002.patch JIRA Issue YARN-5486 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a32d2a14cf8f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1831be8 Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/13229/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13229/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13229/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13229/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13229/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/13229/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13229/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13229/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Uploading new version of patch, fixing compile, unit test and checkstyle issues.

          Show
          kkaranasos Konstantinos Karanasos added a comment - Uploading new version of patch, fixing compile, unit test and checkstyle issues.
          Hide
          asuresh Arun Suresh added a comment -

          Thanks for updating the patch Konstantinos Karanasos

          One possible optimization:

          • Currently, the allocator is called in the same thread as the allocate call. Instead of holding up the AM, this can be done asynchronously and the new containers can be pushed to the SchedulerApplicationAttempt::newlyAllocatedContainers as and when the allocation completes.. but i'm fine handling this in a separate JIRA.

          Couple of minor nits:

          • The TestOpportunisticContainerAllocation has a lot of code in common with TestAMRMClient. I understand it is difficult to extend the later since most of the fields are statically initialized and pulling it out might not be trivial. Can you maybe raise a JIRA to track this?
          • The comment above the DistributedScheduler::updateAllocateResponse() has an extra if which can be removed
          • In ResourceManager, line 1186, the exception message should read be enabled when distr...

          +1 pending the above and a clean Jenkins run.

          Show
          asuresh Arun Suresh added a comment - Thanks for updating the patch Konstantinos Karanasos One possible optimization: Currently, the allocator is called in the same thread as the allocate call. Instead of holding up the AM, this can be done asynchronously and the new containers can be pushed to the SchedulerApplicationAttempt::newlyAllocatedContainers as and when the allocation completes.. but i'm fine handling this in a separate JIRA. Couple of minor nits: The TestOpportunisticContainerAllocation has a lot of code in common with TestAMRMClient . I understand it is difficult to extend the later since most of the fields are statically initialized and pulling it out might not be trivial. Can you maybe raise a JIRA to track this? The comment above the DistributedScheduler::updateAllocateResponse() has an extra if which can be removed In ResourceManager , line 1186, the exception message should read be enabled when distr... +1 pending the above and a clean Jenkins run.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 7m 7s trunk passed
          +1 compile 2m 26s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 55s trunk passed
          +1 mvneclipse 0m 57s trunk passed
          +1 findbugs 2m 52s trunk passed
          +1 javadoc 1m 10s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 34s the patch passed
          +1 compile 2m 24s the patch passed
          +1 javac 2m 24s the patch passed
          -1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn: The patch generated 8 new + 141 unchanged - 0 fixed = 149 total (was 141)
          +1 mvnsite 1m 47s the patch passed
          +1 mvneclipse 0m 50s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 17s the patch passed
          +1 javadoc 1m 0s the patch passed
          +1 unit 0m 26s hadoop-yarn-server-common in the patch passed.
          +1 unit 14m 54s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 34m 49s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 unit 16m 10s hadoop-yarn-client in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          96m 59s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830599/YARN-5486.003.patch
          JIRA Issue YARN-5486
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4b8f7f0d59b7 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2acfb1e
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13233/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13233/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13233/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13233/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13233/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 7m 7s trunk passed +1 compile 2m 26s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 55s trunk passed +1 mvneclipse 0m 57s trunk passed +1 findbugs 2m 52s trunk passed +1 javadoc 1m 10s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 34s the patch passed +1 compile 2m 24s the patch passed +1 javac 2m 24s the patch passed -1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn: The patch generated 8 new + 141 unchanged - 0 fixed = 149 total (was 141) +1 mvnsite 1m 47s the patch passed +1 mvneclipse 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 17s the patch passed +1 javadoc 1m 0s the patch passed +1 unit 0m 26s hadoop-yarn-server-common in the patch passed. +1 unit 14m 54s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 34m 49s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 16m 10s hadoop-yarn-client in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 96m 59s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830599/YARN-5486.003.patch JIRA Issue YARN-5486 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4b8f7f0d59b7 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2acfb1e Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13233/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13233/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13233/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13233/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13233/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Adding new patch.
          Fixed remaining checkstyle issues and Arun Suresh's comments.
          The testcase is not failing locally for me and does not seem related.

          Arun Suresh, I will create JIRAs to track the two issues you mentioned.
          Good point about the opportunistic container allocation. We should make it asynchronous.

          Show
          kkaranasos Konstantinos Karanasos added a comment - Adding new patch. Fixed remaining checkstyle issues and Arun Suresh 's comments. The testcase is not failing locally for me and does not seem related. Arun Suresh , I will create JIRAs to track the two issues you mentioned. Good point about the opportunistic container allocation. We should make it asynchronous.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 7s Maven dependency ordering for branch
          +1 mvninstall 8m 43s trunk passed
          +1 compile 2m 41s trunk passed
          +1 checkstyle 0m 47s trunk passed
          +1 mvnsite 2m 21s trunk passed
          +1 mvneclipse 1m 6s trunk passed
          +1 findbugs 3m 13s trunk passed
          +1 javadoc 1m 6s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 31s the patch passed
          +1 compile 2m 18s the patch passed
          +1 javac 2m 18s the patch passed
          -1 checkstyle 0m 36s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 141 unchanged - 0 fixed = 142 total (was 141)
          +1 mvnsite 1m 43s the patch passed
          +1 mvneclipse 0m 50s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 57s the patch passed
          +1 javadoc 0m 58s the patch passed
          +1 unit 0m 25s hadoop-yarn-server-common in the patch passed.
          +1 unit 15m 11s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 34m 46s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 unit 16m 9s hadoop-yarn-client in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          100m 21s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830811/YARN-5486.004.patch
          JIRA Issue YARN-5486
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 76bdc93c098c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 47f8092
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13243/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13243/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13243/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 7s Maven dependency ordering for branch +1 mvninstall 8m 43s trunk passed +1 compile 2m 41s trunk passed +1 checkstyle 0m 47s trunk passed +1 mvnsite 2m 21s trunk passed +1 mvneclipse 1m 6s trunk passed +1 findbugs 3m 13s trunk passed +1 javadoc 1m 6s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 31s the patch passed +1 compile 2m 18s the patch passed +1 javac 2m 18s the patch passed -1 checkstyle 0m 36s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 141 unchanged - 0 fixed = 142 total (was 141) +1 mvnsite 1m 43s the patch passed +1 mvneclipse 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 57s the patch passed +1 javadoc 0m 58s the patch passed +1 unit 0m 25s hadoop-yarn-server-common in the patch passed. +1 unit 15m 11s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 34m 46s hadoop-yarn-server-resourcemanager in the patch passed. +1 unit 16m 9s hadoop-yarn-client in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 100m 21s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830811/YARN-5486.004.patch JIRA Issue YARN-5486 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 76bdc93c098c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 47f8092 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13243/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13243/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13243/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment -

          Thanks for the updating the patch Konstantinos Karanasos.
          I have linked the JIRAs you created to this.

          +1, will commit this shortly.

          Show
          asuresh Arun Suresh added a comment - Thanks for the updating the patch Konstantinos Karanasos . I have linked the JIRAs you created to this. +1, will commit this shortly.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10516 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10516/)
          YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate (arun suresh: rev 10be45986cdf86a89055065b752959bd6369d54f)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/OpportunisticContainerAllocatorAMService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/scheduler/OpportunisticContainerAllocator.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/scheduler/DistributedScheduler.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/distributed/NodeQueueLoadMonitor.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestOpportunisticContainerAllocation.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/scheduler/OpportunisticContainerContext.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10516 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10516/ ) YARN-5486 . Update OpportunisticContainerAllocatorAMService::allocate (arun suresh: rev 10be45986cdf86a89055065b752959bd6369d54f) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/OpportunisticContainerAllocatorAMService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/scheduler/OpportunisticContainerAllocator.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/scheduler/DistributedScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/distributed/NodeQueueLoadMonitor.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestOpportunisticContainerAllocation.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/scheduler/OpportunisticContainerContext.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
          Hide
          asuresh Arun Suresh added a comment - - edited

          Thanks for the patch Konstantinos Karanasos and for the reviews Subru Krishnan..
          Committed this to trunk.
          Will commit to branch-2 once YARN-5688 is fixed

          Show
          asuresh Arun Suresh added a comment - - edited Thanks for the patch Konstantinos Karanasos and for the reviews Subru Krishnan .. Committed this to trunk. Will commit to branch-2 once YARN-5688 is fixed
          Hide
          asuresh Arun Suresh added a comment -

          Committing this to branch-2. YARN-5688 can be worked on independently

          Show
          asuresh Arun Suresh added a comment - Committing this to branch-2. YARN-5688 can be worked on independently

            People

            • Assignee:
              kkaranasos Konstantinos Karanasos
              Reporter:
              asuresh Arun Suresh
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development