Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3305

AM-Used Resource for leafqueue is wrongly populated if AM ResourceRequest is less than minimumAllocation

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: scheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      For given any ResourceRequest, CS#allocate normalizes request to minimumAllocation if requested memory is less than minimumAllocation.
      But AM-used resource is updated with actual ResourceRequest made by user. This results in AM container allocation more than Max ApplicationMaster Resource.

      This is because AM-Used is updated with actual ResourceRequest made by user while activating the applications. But during allocation of container, ResourceRequest is normalized to minimumAllocation.

      1. 0001-YARN-3305.patch
        4 kB
        Rohith Sharma K S
      2. 0001-YARN-3305.patch
        4 kB
        Rohith Sharma K S
      3. 0002-YARN-3305.patch
        7 kB
        Rohith Sharma K S
      4. 0003-YARN-3305.patch
        7 kB
        Rohith Sharma K S

        Issue Links

          Activity

          Hide
          rohithsharma Rohith Sharma K S added a comment -

          ResourceRequest's are normalized when CS#allocate has been invoked. But AMUsed is updated while activating the applications which is earlier to CS#allocate call.
          For AM ResourceRequest, normization should be done while adding attempt only.

          Show
          rohithsharma Rohith Sharma K S added a comment - ResourceRequest's are normalized when CS#allocate has been invoked. But AMUsed is updated while activating the applications which is earlier to CS#allocate call. For AM ResourceRequest, normization should be done while adding attempt only.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Updated the patch for normalizing ResourceRequest when there is attempt is added. Kindly review the pach.

          Show
          rohithsharma Rohith Sharma K S added a comment - Updated the patch for normalizing ResourceRequest when there is attempt is added. Kindly review the pach.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12703602/0001-YARN-3305.patch
          against trunk revision 7711049.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 5 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6900//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6900//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6900//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703602/0001-YARN-3305.patch against trunk revision 7711049. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 5 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6900//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6900//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6900//console This message is automatically generated.
          Hide
          jianhe Jian He added a comment -

          Rohith Sharma K S, thanks for working on this !
          I think we can call normalize the am request in RMAppManager after validateAndCreateResourceRequest to make sure the am request stored in RMAppIml is also correct.

          Show
          jianhe Jian He added a comment - Rohith Sharma K S , thanks for working on this ! I think we can call normalize the am request in RMAppManager after validateAndCreateResourceRequest to make sure the am request stored in RMAppIml is also correct.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          call normalize the am request in RMAppManager after validateAndCreateResourceRequest to make sure the am request stored in RMAppIml is also correct.

          Agree.. Will change it

          Show
          rohithsharma Rohith Sharma K S added a comment - call normalize the am request in RMAppManager after validateAndCreateResourceRequest to make sure the am request stored in RMAppIml is also correct. Agree.. Will change it
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Updated patch as per comment.. Kindly review that latest patch

          Show
          rohithsharma Rohith Sharma K S added a comment - Updated patch as per comment.. Kindly review that latest patch
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12704352/0001-YARN-3305.patch
          against trunk revision a852910.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 5 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
          org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
          org.apache.hadoop.yarn.server.resourcemanager.TestAppManager
          org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6950//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6950//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6950//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704352/0001-YARN-3305.patch against trunk revision a852910. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 5 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate org.apache.hadoop.yarn.server.resourcemanager.TestAppManager org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6950//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6950//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6950//console This message is automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Updated the patch correcting test failures. TestCapacitySchedulerNodeLabelUpdate failure JIRA is YARN-3343 .
          Kindly review the updated patch

          Show
          rohithsharma Rohith Sharma K S added a comment - Updated the patch correcting test failures. TestCapacitySchedulerNodeLabelUpdate failure JIRA is YARN-3343 . Kindly review the updated patch
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12704406/0002-YARN-3305.patch
          against trunk revision 387f271.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 5 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestRM

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6955//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6955//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6955//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704406/0002-YARN-3305.patch against trunk revision 387f271. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 5 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestRM Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6955//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6955//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6955//console This message is automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Failed test is unrelated to this patch.

          Show
          rohithsharma Rohith Sharma K S added a comment - Failed test is unrelated to this patch.
          Hide
          jianhe Jian He added a comment -

          looks good overall, why is unmanagedAM check needed?

          if (!submissionContext.getUnmanagedAM())
          
          Show
          jianhe Jian He added a comment - looks good overall, why is unmanagedAM check needed? if (!submissionContext.getUnmanagedAM())
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Unmanaged applications need not necessarily send AM RR(ResourceRequest) because RM won't allocate a container for the AM and start it. Instead RM expect AM to be launched and connect to the RM within the AM liveliness period. So for unmanaged applications RR can be null which would cause NPE while normalizing RR.

          Show
          rohithsharma Rohith Sharma K S added a comment - Unmanaged applications need not necessarily send AM RR(ResourceRequest) because RM won't allocate a container for the AM and start it. Instead RM expect AM to be launched and connect to the RM within the AM liveliness period. So for unmanaged applications RR can be null which would cause NPE while normalizing RR.
          Hide
          jianhe Jian He added a comment -

          ah, right. forgot about that.
          will commit this. thanks !

          Show
          jianhe Jian He added a comment - ah, right. forgot about that. will commit this. thanks !
          Hide
          jianhe Jian He added a comment -

          one minor thing, given the previous validateAndCreateResourceRequest has the same check. do you think we can move the normalizeRequest into that method ?

          Show
          jianhe Jian He added a comment - one minor thing, given the previous validateAndCreateResourceRequest has the same check. do you think we can move the normalizeRequest into that method ?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Right, i was not seen this check. It can be moved to inside validateAndCreateResourceRequest.

          Show
          rohithsharma Rohith Sharma K S added a comment - Right, i was not seen this check. It can be moved to inside validateAndCreateResourceRequest.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Attached updated patch, kindly review

          Show
          rohithsharma Rohith Sharma K S added a comment - Attached updated patch, kindly review
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12704939/0003-YARN-3305.patch
          against trunk revision 5608520.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 5 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6990//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6990//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6990//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704939/0003-YARN-3305.patch against trunk revision 5608520. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 5 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6990//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6990//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6990//console This message is automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Test failure is unrelated to this patch

          Show
          rohithsharma Rohith Sharma K S added a comment - Test failure is unrelated to this patch
          Hide
          jianhe Jian He added a comment -

          Committed to trunk and branch-2, thanks Rohith !

          Show
          jianhe Jian He added a comment - Committed to trunk and branch-2, thanks Rohith !
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7352 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7352/)
          YARN-3305. Normalize AM resource request on app submission. Contributed by Rohith Sharmaks (jianhe: rev 968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7352 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7352/ ) YARN-3305 . Normalize AM resource request on app submission. Contributed by Rohith Sharmaks (jianhe: rev 968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #136 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/136/)
          YARN-3305. Normalize AM resource request on app submission. Contributed by Rohith Sharmaks (jianhe: rev 968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #136 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/136/ ) YARN-3305 . Normalize AM resource request on app submission. Contributed by Rohith Sharmaks (jianhe: rev 968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #870 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/870/)
          YARN-3305. Normalize AM resource request on app submission. Contributed by Rohith Sharmaks (jianhe: rev 968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #870 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/870/ ) YARN-3305 . Normalize AM resource request on app submission. Contributed by Rohith Sharmaks (jianhe: rev 968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2068 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2068/)
          YARN-3305. Normalize AM resource request on app submission. Contributed by Rohith Sharmaks (jianhe: rev 968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2068 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2068/ ) YARN-3305 . Normalize AM resource request on app submission. Contributed by Rohith Sharmaks (jianhe: rev 968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #127 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/127/)
          YARN-3305. Normalize AM resource request on app submission. Contributed by Rohith Sharmaks (jianhe: rev 968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #127 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/127/ ) YARN-3305 . Normalize AM resource request on app submission. Contributed by Rohith Sharmaks (jianhe: rev 968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2086 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2086/)
          YARN-3305. Normalize AM resource request on app submission. Contributed by Rohith Sharmaks (jianhe: rev 968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2086 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2086/ ) YARN-3305 . Normalize AM resource request on app submission. Contributed by Rohith Sharmaks (jianhe: rev 968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #136 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/136/)
          YARN-3305. Normalize AM resource request on app submission. Contributed by Rohith Sharmaks (jianhe: rev 968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #136 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/136/ ) YARN-3305 . Normalize AM resource request on app submission. Contributed by Rohith Sharmaks (jianhe: rev 968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user smarella commented on the pull request:

          https://github.com/apache/hadoop/commit/968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf#commitcomment-14835835

          In hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java:
          In hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java on line 397:
          YARN-3305 seem to break the Myriad builds (see below). As indicated in YARN-3305, https://issues.apache.org/jira/browse/YARN-3996 is trying to fix the problem. YARN-3996 is currently work-in-progress, but we need to back port some of it immediately to unblock Myriad. Once YARN-3996 is fully fixed, it needs to be fully back ported. I've requested Sarjeet to raise two bugs for this (one to provide an immediate fix and second to back port YARN-3996 once it's resolved).

          Currently, the latest Myriad builds are effected by this change. The problem is that, the AM container resources are normalized during app submission with ```scheduler.getminimumResourceCapability()```, which comes from yarn.scheduler.minimum-allocation-

          {mb,vcores,disks}. These values are set to "0" by Myriad. Hence, the AM container allocated will be of zero size and is killed by NM as soon as it is launched.

          The reason why Myriad needs yarn.scheduler.minimum-allocation-{mb,vcores,disks}

          to be 0 is for discussed in https://issues.apache.org/jira/browse/MYRIAD-139.

          Show
          githubbot ASF GitHub Bot added a comment - Github user smarella commented on the pull request: https://github.com/apache/hadoop/commit/968425e9f7b850ff9c2ab8ca37a64c3fdbe77dbf#commitcomment-14835835 In hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java: In hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java on line 397: YARN-3305 seem to break the Myriad builds (see below). As indicated in YARN-3305 , https://issues.apache.org/jira/browse/YARN-3996 is trying to fix the problem. YARN-3996 is currently work-in-progress, but we need to back port some of it immediately to unblock Myriad. Once YARN-3996 is fully fixed, it needs to be fully back ported. I've requested Sarjeet to raise two bugs for this (one to provide an immediate fix and second to back port YARN-3996 once it's resolved). Currently, the latest Myriad builds are effected by this change. The problem is that, the AM container resources are normalized during app submission with ```scheduler.getminimumResourceCapability()```, which comes from yarn.scheduler.minimum-allocation- {mb,vcores,disks}. These values are set to "0" by Myriad. Hence, the AM container allocated will be of zero size and is killed by NM as soon as it is launched. The reason why Myriad needs yarn.scheduler.minimum-allocation-{mb,vcores,disks} to be 0 is for discussed in https://issues.apache.org/jira/browse/MYRIAD-139 .

            People

            • Assignee:
              rohithsharma Rohith Sharma K S
              Reporter:
              rohithsharma Rohith Sharma K S
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development