Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2492 (Clone of YARN-796) Allow for (admin) labels on nodes and resource-requests
  3. YARN-2694

Ensure only single node labels specified in resource request / host, and node label expression only specified when resourceName=ANY

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Currently, node label expression supporting in capacity scheduler is partial completed. Now node label expression specified in Resource Request will only respected when it specified at ANY level. And a ResourceRequest/host with multiple node labels will make user limit, etc. computation becomes more tricky.

      Now we need temporarily disable them, changes include,

      • AMRMClient
      • ApplicationMasterService
      • RMAdminCLI
      • CommonNodeLabelsManager
      1. YARN-2694-branch-2.6.1.txt
        50 kB
        Vinod Kumar Vavilapalli
      2. YARN-2694-20150205-3.patch
        59 kB
        Wangda Tan
      3. YARN-2694-20150205-2.patch
        58 kB
        Wangda Tan
      4. YARN-2694-20150205-1.patch
        59 kB
        Wangda Tan
      5. YARN-2694-20150204-1.patch
        58 kB
        Wangda Tan
      6. YARN-2694-20150203-2.patch
        57 kB
        Wangda Tan
      7. YARN-2694-20150203-1.patch
        56 kB
        Wangda Tan
      8. YARN-2694-20150202-1.patch
        50 kB
        Wangda Tan
      9. YARN-2694-20150122-1.patch
        51 kB
        Wangda Tan
      10. YARN-2694-20150121-1.patch
        42 kB
        Wangda Tan
      11. YARN-2694-20141101-2.patch
        43 kB
        Wangda Tan
      12. YARN-2694-20141101-1.patch
        42 kB
        Wangda Tan
      13. YARN-2694-20141023-2.patch
        42 kB
        Wangda Tan
      14. YARN-2694-20141023-1.patch
        42 kB
        Wangda Tan
      15. YARN-2694-20141021-1.patch
        23 kB
        Wangda Tan
      16. YARN-2694-20141020-1.patch
        22 kB
        Wangda Tan

        Issue Links

          Activity

          Hide
          zhengchenyu zhengchenyu added a comment - - edited

          Naganarasimha G RThanks for your explaination, I know this. In our modification, we doesn't consider the lables as a resource pool, so we didn't set the resource for label. So we open the multi-nodelabel.

          Show
          zhengchenyu zhengchenyu added a comment - - edited Naganarasimha G R Thanks for your explaination, I know this. In our modification, we doesn't consider the lables as a resource pool, so we didn't set the resource for label. So we open the multi-nodelabel.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi zhengchenyu,
          Currently labels are viewed as a partition or pool where resource guarantee is ensured among the queues who have the access to it.
          So say cluster has a single node and node has labels x,y. There exists 2 queues Q1 and Q2, and for x label guarantees are Q1(25) & Q2(75), for y label guarantees are Q1(50) & Q2(50), suppose for some app in Q1 requesting for label x is already allocated 25% and now some other app in Q1 is requesting for label y then do you need to allocate this node to the app in Q1 ? if you allocate then guarantees for label x for the Q2 will not be met.
          Practically a node can belong to only one partition when we want to ensure some policy of resource sharing among the queues (irrespective of FAIR/CS ). Hence the above modification was done. To support multiple labels(precisely node attributes) we have raised YARN-3409. May be more can be discussed there.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi zhengchenyu , Currently labels are viewed as a partition or pool where resource guarantee is ensured among the queues who have the access to it. So say cluster has a single node and node has labels x,y. There exists 2 queues Q1 and Q2, and for x label guarantees are Q1(25) & Q2(75), for y label guarantees are Q1(50) & Q2(50), suppose for some app in Q1 requesting for label x is already allocated 25% and now some other app in Q1 is requesting for label y then do you need to allocate this node to the app in Q1 ? if you allocate then guarantees for label x for the Q2 will not be met. Practically a node can belong to only one partition when we want to ensure some policy of resource sharing among the queues (irrespective of FAIR/CS ). Hence the above modification was done. To support multiple labels(precisely node attributes) we have raised YARN-3409 . May be more can be discussed there.
          Hide
          zhengchenyu zhengchenyu added a comment - - edited

          I don't know why disabled the function of supporting multi-nodelabel for nodemanager. Maybe your worry about the complex of calculating the nodelables's resource?
          We add the nodelabel to fairscheduler, but we dosen't add the calculation for nodelabel's resource. So I think there is no need to disable this funciton for us.
          Can you give me some advice or some historical reason for us ?

          Show
          zhengchenyu zhengchenyu added a comment - - edited I don't know why disabled the function of supporting multi-nodelabel for nodemanager. Maybe your worry about the complex of calculating the nodelables's resource? We add the nodelabel to fairscheduler, but we dosen't add the calculation for nodelabel's resource. So I think there is no need to disable this funciton for us. Can you give me some advice or some historical reason for us ?
          Hide
          zhz Zhe Zhang added a comment -

          Thanks a lot Wangda for the clear explanation! I think YARN-4140 is what we need.

          Show
          zhz Zhe Zhang added a comment - Thanks a lot Wangda for the clear explanation! I think YARN-4140 is what we need.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Zhe Zhang,

          The major reason of only supporting specify node label in off-switch request is: existing node label is actually node partition, admin can limit how much resource that can be used by each queue in each partition. So if rack / node level request has different partition of off-switch request, scheduler cannot correctly calculate how much resource the app requested in each partition: all existing logic to calculate pending resources of an app relies on off-switch request.

          This patch doesn't allow specify any label in rack / node, all rack / node request contains label will be directly rejected. This is not correct, we fixed this in YARN-4140 that forces all requests under the off-switch request has same label. You can take a look at discussions on YARN-4140.

          If you want rack / node specify different label, they will be available after YARN-3409 (node constraints). Node constraints will be used to tag NMs, no more enforcements (such as ACL, resource limtis, etc.) will be added to node constraints.

          Show
          leftnoteasy Wangda Tan added a comment - Zhe Zhang , The major reason of only supporting specify node label in off-switch request is: existing node label is actually node partition, admin can limit how much resource that can be used by each queue in each partition. So if rack / node level request has different partition of off-switch request, scheduler cannot correctly calculate how much resource the app requested in each partition: all existing logic to calculate pending resources of an app relies on off-switch request. This patch doesn't allow specify any label in rack / node, all rack / node request contains label will be directly rejected. This is not correct, we fixed this in YARN-4140 that forces all requests under the off-switch request has same label. You can take a look at discussions on YARN-4140 . If you want rack / node specify different label, they will be available after YARN-3409 (node constraints). Node constraints will be used to tag NMs, no more enforcements (such as ACL, resource limtis, etc.) will be added to node constraints.
          Hide
          zhz Zhe Zhang added a comment -

          Hi Jian He, Wangda Tan, I have a few questions about this change:

          Currently, node label expression supporting in capacity scheduler is partial completed. Now node label expression specified in Resource Request will only respected when it specified at ANY level.

          Could you elaborate a bit on this? Is it because it's hard to satisfy both node label and locality requirements? If so, when do you think we will be ready to enable Node or Rack level resource requests?

          Thanks,

          Show
          zhz Zhe Zhang added a comment - Hi Jian He , Wangda Tan , I have a few questions about this change: Currently, node label expression supporting in capacity scheduler is partial completed. Now node label expression specified in Resource Request will only respected when it specified at ANY level. Could you elaborate a bit on this? Is it because it's hard to satisfy both node label and locality requirements? If so, when do you think we will be ready to enable Node or Rack level resource requests? Thanks,
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Attaching patch that I committed to 2.6.1.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Attaching patch that I committed to 2.6.1.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Pulled this into 2.6.1 as a dependency.

          Had to fix merge conflicts. Test changes to several tests don't apply as they don't exist in 2.6.1. Similarly some changes in TestRMRestart don't apply. So dropped them.

          Ran all ResourceManager tests before pushing this in.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Pulled this into 2.6.1 as a dependency. Had to fix merge conflicts. Test changes to several tests don't apply as they don't exist in 2.6.1. Similarly some changes in TestRMRestart don't apply. So dropped them. Ran all ResourceManager tests before pushing this in.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #99 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/99/)

          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #99 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/99/ )
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2049 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2049/)

          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2049 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2049/ )
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #95 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/95/)

          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #95 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/95/ )
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2030 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2030/)

          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2030 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2030/ )
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #98 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/98/)

          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #98 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/98/ )
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #832 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/832/)

          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #832 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/832/ )
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7042 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7042/)
          YARN-2694. Ensure only single node label specified in ResourceRequest. Contributed by Wangda Tan (jianhe: rev c1957fef29b07fea70938e971b30532a1e131fd0)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMNodeLabelsManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestCommonNodeLabelsManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7042 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7042/ ) YARN-2694 . Ensure only single node label specified in ResourceRequest. Contributed by Wangda Tan (jianhe: rev c1957fef29b07fea70938e971b30532a1e131fd0) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMNodeLabelsManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestCommonNodeLabelsManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java
          Hide
          jianhe Jian He added a comment -

          Committed to trunk and branch-2, thanks Wangda !

          Show
          jianhe Jian He added a comment - Committed to trunk and branch-2, thanks Wangda !
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12696916/YARN-2694-20150205-3.patch
          against trunk revision 4641196.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
          org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6532//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6532//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696916/YARN-2694-20150205-3.patch against trunk revision 4641196. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.client.TestResourceTrackerOnHA org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6532//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6532//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Reverted change of

                  if (null == req.getNodeLabelExpression()
                      && ResourceRequest.ANY.equals(req.getResourceName())) {
                    req.setNodeLabelExpression(asc.getNodeLabelExpression());
                  }
          

          In ApplicationMasterService

          We should not check empty here, because empty means it has being set. And this logic is "overwrite when not set", updated patch.

          Show
          leftnoteasy Wangda Tan added a comment - Reverted change of if ( null == req.getNodeLabelExpression() && ResourceRequest.ANY.equals(req.getResourceName())) { req.setNodeLabelExpression(asc.getNodeLabelExpression()); } In ApplicationMasterService We should not check empty here, because empty means it has being set. And this logic is "overwrite when not set", updated patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12696888/YARN-2694-20150205-2.patch
          against trunk revision e1990ab.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
          org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6529//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6529//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696888/YARN-2694-20150205-2.patch against trunk revision e1990ab. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.client.TestResourceTrackerOnHA org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6529//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6529//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Local repo is missed, regenerated patch.

          Show
          leftnoteasy Wangda Tan added a comment - Local repo is missed, regenerated patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12696874/YARN-2694-20150205-1.patch
          against trunk revision e1990ab.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6527//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696874/YARN-2694-20150205-1.patch against trunk revision e1990ab. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6527//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Jian He,
          Thanks for your comments,
          Addressed most of them, except:

          Instead of throwing IOException, use precondition check too

          It will throw RuntimeException, I'm not sure will it be caught by framework.

          maybe check “||” too ?

          It's no need to check "||" here, it is already sufficient, because an empty exp will not contain "&&"

          Attached new patch.

          Show
          leftnoteasy Wangda Tan added a comment - Jian He , Thanks for your comments, Addressed most of them, except: Instead of throwing IOException, use precondition check too It will throw RuntimeException, I'm not sure will it be caught by framework. maybe check “||” too ? It's no need to check "||" here, it is already sufficient, because an empty exp will not contain "&&" Attached new patch.
          Hide
          jianhe Jian He added a comment -

          few minor comments, looks good overall,

          • check if empty too ?
                if (null == exp) {
                  return;
                }
            
          • remove the IOException from the method header
              private Map<NodeId, Set<String>> buildNodeLabelsMapFromStr(String args)
                  throws IOException 
            
          • exceed 80 column limit
            private void verifyAddRequestFailed(AMRMClient<ContainerRequest> client, ContainerRequest request) {
            
          • Instead of throwing IOException, use precondition check too
            if (labels.size() > 1) {
                    String msg = String.format("%d labels specified on host=%s"
                        + ", please note that we do not support specifying multiple"
                        + " labels on a single host for now.", labels.size(),
                        nodeId.getHost());
                    LOG.error(msg);
                    
            
          • should the following check empty string too ?
             if (null == req.getNodeLabelExpression()
                        && ResourceRequest.ANY.equals(req.getResourceName())) {
            
          • maybe check “||” too ?
            if (labelExp != null && labelExp.contains("&&")) {
            
          Show
          jianhe Jian He added a comment - few minor comments, looks good overall, check if empty too ? if ( null == exp) { return ; } remove the IOException from the method header private Map<NodeId, Set< String >> buildNodeLabelsMapFromStr( String args) throws IOException exceed 80 column limit private void verifyAddRequestFailed(AMRMClient<ContainerRequest> client, ContainerRequest request) { Instead of throwing IOException, use precondition check too if (labels.size() > 1) { String msg = String .format( "%d labels specified on host=%s" + ", please note that we do not support specifying multiple" + " labels on a single host for now." , labels.size(), nodeId.getHost()); LOG.error(msg); should the following check empty string too ? if ( null == req.getNodeLabelExpression() && ResourceRequest.ANY.equals(req.getResourceName())) { maybe check “||” too ? if (labelExp != null && labelExp.contains( "&&" )) {
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12696524/YARN-2694-20150204-1.patch
          against trunk revision 064e077.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
          org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6510//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6510//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696524/YARN-2694-20150204-1.patch against trunk revision 064e077. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.client.TestResourceTrackerOnHA org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6510//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6510//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12696352/YARN-2694-20150203-2.patch
          against trunk revision e5b56e2.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified test files.

          -1 javac. The applied patch generated 1151 javac compiler warnings (more than the trunk's current 1150 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
          org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
          org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
          org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6502//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6502//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6502//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696352/YARN-2694-20150203-2.patch against trunk revision e5b56e2. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. -1 javac . The applied patch generated 1151 javac compiler warnings (more than the trunk's current 1150 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.client.TestResourceTrackerOnHA org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6502//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6502//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6502//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          I think it's better to do it in a separated ticket or we will completely mix them together.

          Show
          leftnoteasy Wangda Tan added a comment - I think it's better to do it in a separated ticket or we will completely mix them together.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Hi Jian He,
          Thanks for your comments, addressed previously 4, and for the last one:

          this should be after request sanity validation

          Since the sanity validation depends on previously set label-expression, so it's not correct to move it to below sanity check.

          Show
          leftnoteasy Wangda Tan added a comment - Hi Jian He , Thanks for your comments, addressed previously 4, and for the last one: this should be after request sanity validation Since the sanity validation depends on previously set label-expression, so it's not correct to move it to below sanity check.
          Hide
          jianhe Jian He added a comment -

          one other comment: revert changes regarding checkAddLabelsToNode, given this method is currently not used at all. If it's used, it means we need to support multiple labels per node, meaning we need to revert this change again including the tests.

          Show
          jianhe Jian He added a comment - one other comment: revert changes regarding checkAddLabelsToNode, given this method is currently not used at all. If it's used, it means we need to support multiple labels per node, meaning we need to revert this change again including the tests.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12696314/YARN-2694-20150203-1.patch
          against trunk revision b73e776.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified test files.

          -1 javac. The applied patch generated 1151 javac compiler warnings (more than the trunk's current 1150 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6501//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6501//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6501//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696314/YARN-2694-20150203-1.patch against trunk revision b73e776. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. -1 javac . The applied patch generated 1151 javac compiler warnings (more than the trunk's current 1150 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6501//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6501//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6501//console This message is automatically generated.
          Hide
          jianhe Jian He added a comment -
          • check “||” too? “more than two node labels”, two is also invalid. call it multiple node labels.
            // Don't support specifying >= 2 node labels in a node label expression now
                if (null != exp && exp.contains("&&")) {
                  throw new InvalidContainerRequestException(
                      "Cannot specify more than two node labels"
                          + " in a single node label expression");
                }
            
            
          • for above, also throw exception if node label is specified for a non-any request.
          • should not use IOException? throw new IOException(String.format("%d labels specified on host=%s”, we may use the Precondition.check
          • may be say: Note that we currently do not support specifying multiple labels on a single host?
            please note we temporarily disable specify"
                        + " more than one label on each host for now.
            
          • this should be after request sanity validation
                  // set label expression for Resource Requests if resourceName=ANY 
                  ApplicationSubmissionContext asc = app.getApplicationSubmissionContext();
                  for (ResourceRequest req : ask) {
                    if (null == req.getNodeLabelExpression()
                        && ResourceRequest.ANY.equals(req.getResourceName())) {
                      req.setNodeLabelExpression(asc.getNodeLabelExpression());
                    }
                  }
            
          Show
          jianhe Jian He added a comment - check “||” too? “more than two node labels”, two is also invalid. call it multiple node labels. // Don't support specifying >= 2 node labels in a node label expression now if ( null != exp && exp.contains( "&&" )) { throw new InvalidContainerRequestException( "Cannot specify more than two node labels" + " in a single node label expression" ); } for above, also throw exception if node label is specified for a non-any request. should not use IOException? throw new IOException(String.format("%d labels specified on host=%s” , we may use the Precondition.check may be say: Note that we currently do not support specifying multiple labels on a single host? please note we temporarily disable specify" + " more than one label on each host for now. this should be after request sanity validation // set label expression for Resource Requests if resourceName=ANY ApplicationSubmissionContext asc = app.getApplicationSubmissionContext(); for (ResourceRequest req : ask) { if ( null == req.getNodeLabelExpression() && ResourceRequest.ANY.equals(req.getResourceName())) { req.setNodeLabelExpression(asc.getNodeLabelExpression()); } }
          Hide
          leftnoteasy Wangda Tan added a comment -

          Rebased against trunk and resolved javac warning.

          Show
          leftnoteasy Wangda Tan added a comment - Rebased against trunk and resolved javac warning.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12696025/YARN-2694-20150202-1.patch
          against trunk revision bd69fb2.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6499//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696025/YARN-2694-20150202-1.patch against trunk revision bd69fb2. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6499//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12696025/YARN-2694-20150202-1.patch
          against trunk revision 8acc5e9.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified test files.

          -1 javac. The applied patch generated 1189 javac compiler warnings (more than the trunk's current 1187 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
          org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
          org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6485//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6485//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6485//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696025/YARN-2694-20150202-1.patch against trunk revision 8acc5e9. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. -1 javac . The applied patch generated 1189 javac compiler warnings (more than the trunk's current 1187 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.client.TestResourceTrackerOnHA org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6485//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6485//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6485//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Rebased against trunk

          Show
          leftnoteasy Wangda Tan added a comment - Rebased against trunk
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12693971/YARN-2694-20150122-1.patch
          against trunk revision 825923f.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified test files.

          -1 javac. The applied patch generated 1206 javac compiler warnings (more than the trunk's current 1204 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
          org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6391//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6391//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6391//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693971/YARN-2694-20150122-1.patch against trunk revision 825923f. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. -1 javac . The applied patch generated 1206 javac compiler warnings (more than the trunk's current 1204 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.client.TestResourceTrackerOnHA org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6391//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6391//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6391//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Updated patch addressed tests failures.

          Show
          leftnoteasy Wangda Tan added a comment - Updated patch addressed tests failures.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12693762/YARN-2694-20150121-1.patch
          against trunk revision ee7d22e.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          -1 javac. The applied patch generated 1206 javac compiler warnings (more than the trunk's current 1204 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
          org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
          org.apache.hadoop.yarn.client.cli.TestRMAdminCLI
          org.apache.hadoop.yarn.nodelabels.TestCommonNodeLabelsManager
          org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels
          org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6383//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6383//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6383//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693762/YARN-2694-20150121-1.patch against trunk revision ee7d22e. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. -1 javac . The applied patch generated 1206 javac compiler warnings (more than the trunk's current 1204 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.client.TestResourceTrackerOnHA org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA org.apache.hadoop.yarn.client.cli.TestRMAdminCLI org.apache.hadoop.yarn.nodelabels.TestCommonNodeLabelsManager org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6383//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6383//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6383//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Rebased against latest trunk

          Show
          leftnoteasy Wangda Tan added a comment - Rebased against latest trunk
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12678768/YARN-2694-20141101-2.patch
          against trunk revision 5c0381c.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
          org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5683//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5683//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12678768/YARN-2694-20141101-2.patch against trunk revision 5c0381c. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.client.TestResourceTrackerOnHA org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5683//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5683//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Updated patch fixed failed tests

          Show
          leftnoteasy Wangda Tan added a comment - Updated patch fixed failed tests
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12678692/YARN-2694-20141101-1.patch
          against trunk revision ed63b11.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.client.cli.TestRMAdminCLI

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5677//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5677//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12678692/YARN-2694-20141101-1.patch against trunk revision ed63b11. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.client.cli.TestRMAdminCLI +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5677//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5677//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Rebased against latest trunk

          Show
          leftnoteasy Wangda Tan added a comment - Rebased against latest trunk
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12676740/YARN-2694-20141023-2.patch
          against trunk revision db45f04.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5534//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5534//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676740/YARN-2694-20141023-2.patch against trunk revision db45f04. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5534//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5534//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12676740/YARN-2694-20141023-2.patch
          against trunk revision 828429d.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5526//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5526//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676740/YARN-2694-20141023-2.patch against trunk revision 828429d. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5526//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5526//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          I can compile this locally, I haven't found any error message in the console log of Jenkins result. So I suspect it just Jenkins process crashed. Resubmit same patch.

          Show
          leftnoteasy Wangda Tan added a comment - I can compile this locally, I haven't found any error message in the console log of Jenkins result. So I suspect it just Jenkins process crashed. Resubmit same patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12676721/YARN-2694-20141023-1.patch
          against trunk revision 828429d.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5523//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676721/YARN-2694-20141023-1.patch against trunk revision 828429d. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5523//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Updated patch

          Show
          leftnoteasy Wangda Tan added a comment - Updated patch
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12676130/YARN-2694-20141021-1.patch
          against trunk revision 6f2028b.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5484//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5484//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676130/YARN-2694-20141021-1.patch against trunk revision 6f2028b. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5484//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5484//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Attached updated patch fixed test failure of TestContainerAllocation

          Show
          leftnoteasy Wangda Tan added a comment - Attached updated patch fixed test failure of TestContainerAllocation
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12675967/YARN-2694-20141020-1.patch
          against trunk revision e90718f.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5471//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5471//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12675967/YARN-2694-20141020-1.patch against trunk revision e90718f. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5471//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5471//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Attached ver.1 patch and kick Jenkins

          Show
          leftnoteasy Wangda Tan added a comment - Attached ver.1 patch and kick Jenkins

            People

            • Assignee:
              leftnoteasy Wangda Tan
              Reporter:
              leftnoteasy Wangda Tan
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development