Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: scheduler
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Trying to run application on Nodelabel partition I found that the application execution time is delayed by 5 – 10 min for 500 containers . Total 3 machines 2 machines were in same partition and app submitted to same.

      After enabling debug was able to find the below

      1. From AM the container ask is for OFF-SWITCH
      2. RM allocating all containers to NODE_LOCAL as shown in logs below.
      3. So since I was having about 500 containers time taken was about – 6 minutes to allocate 1st map after AM allocation.
      4. Tested with about 1K maps using PI job took 17 minutes to allocate next container after AM allocation

      Once 500 container allocation on NODE_LOCAL is done the next container allocation is done on OFF_SWITCH

      2015-09-09 15:21:58,954 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt: showRequests: application=application_1441791998224_0001 request={Priority: 20, Capability: <memory:512, vCores:1>, # Containers: 500, Location: /default-rack, Relax Locality: true, Node Label Expression: }
      
      2015-09-09 15:21:58,954 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt: showRequests: application=application_1441791998224_0001 request={Priority: 20, Capability: <memory:512, vCores:1>, # Containers: 500, Location: *, Relax Locality: true, Node Label Expression: 3}
      
      2015-09-09 15:21:58,954 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt: showRequests: application=application_1441791998224_0001 request={Priority: 20, Capability: <memory:512, vCores:1>, # Containers: 500, Location: host-10-19-92-143, Relax Locality: true, Node Label Expression: }
      
      2015-09-09 15:21:58,954 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt: showRequests: application=application_1441791998224_0001 request={Priority: 20, Capability: <memory:512, vCores:1>, # Containers: 500, Location: host-10-19-92-117, Relax Locality: true, Node Label Expression: }
      
      2015-09-09 15:21:58,954 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> <memory:0, vCores:0>, NODE_LOCAL
      
      2015-09-09 14:35:45,467 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> <memory:0, vCores:0>, NODE_LOCAL
      
      2015-09-09 14:35:45,831 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> <memory:0, vCores:0>, NODE_LOCAL
      
      2015-09-09 14:35:46,469 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> <memory:0, vCores:0>, NODE_LOCAL
      
      2015-09-09 14:35:46,832 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Assigned to queue: root.b.b1 stats: b1: capacity=1.0, absoluteCapacity=0.5, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=1 --> <memory:0, vCores:0>, NODE_LOCAL
      
      
      dsperf@host-127:/opt/bibin/dsperf/HAINSTALL/install/hadoop/resourcemanager/logs1> cat hadoop-dsperf-resourcemanager-host-127.log | grep "NODE_LOCAL" | grep "root.b.b1" | wc -l
      
      500
      

      (Consumes about 6 minutes)

      1. 0001-YARN-4140.patch
        3 kB
        Bibin A Chundatt
      2. 0002-YARN-4140.patch
        3 kB
        Bibin A Chundatt
      3. 0003-YARN-4140.patch
        7 kB
        Bibin A Chundatt
      4. 0004-YARN-4140.patch
        10 kB
        Bibin A Chundatt
      5. 0005-YARN-4140.patch
        8 kB
        Bibin A Chundatt
      6. 0006-YARN-4140.patch
        8 kB
        Bibin A Chundatt
      7. 0007-YARN-4140.patch
        9 kB
        Bibin A Chundatt
      8. 0008-YARN-4140.patch
        12 kB
        Bibin A Chundatt
      9. 0009-YARN-4140.patch
        13 kB
        Bibin A Chundatt
      10. 0010-YARN-4140.patch
        12 kB
        Bibin A Chundatt
      11. 0011-YARN-4140.patch
        12 kB
        Bibin A Chundatt
      12. 0012-YARN-4140.patch
        13 kB
        Bibin A Chundatt
      13. 0013-YARN-4140.patch
        13 kB
        Bibin A Chundatt
      14. 0014-YARN-4140.patch
        17 kB
        Bibin A Chundatt

        Issue Links

          Activity

          Hide
          sunilg Sunil G added a comment -

          Hi Bibin A Chundatt
          Thanks for reporting this issue. I think this has to be handled.
          I think in assignContainer, we must also check for NodeLocal and RackLocal against node_partition label. Now only offrack issues are compared i think.

          Show
          sunilg Sunil G added a comment - Hi Bibin A Chundatt Thanks for reporting this issue. I think this has to be handled. I think in assignContainer , we must also check for NodeLocal and RackLocal against node_partition label. Now only offrack issues are compared i think.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks for reporting this issue Bibin A Chundatt, also did some tests locally. Sunil G mentioned the root cause.

          Currently, we only have nodeLabelExpression assigned to off-switch request, and nodeLabelExpression of node-local/rack-local request is always empty. So SchedulerUtils.checkResourceRequestMatchingNodePartition will be always failed for node-local/rack-local requests.

          And now, computation of off-switch delay has some issues, wait time is linear correlated to #requested-container / #hosts. (Will file ticket to fix that problem.) So in worst cases, scheduler will wait for more than several minutes to get the first off-switch container.

          In this JIRA, I think we need to fix the first problem, we don't allow to set nodeLabelExpression for node-local/rack-local request after YARN-2694, but I think we need to make nodeLabelExpression of node-local/rack-local consistent with off-switch requests.

          Back to the code, I think what need to fix is, in AppSchedulingInfo#updateResourceRequests.

          • When (request.resourceName == ANY && request.node-label-expression updated) we need to update all resource requests of this priority
          • When (request.resourceName != ANY) we need to set request.node-label-expression equals to ANY-resource-request.node-label-expression of the same priority

          Thoughts?

          Show
          leftnoteasy Wangda Tan added a comment - Thanks for reporting this issue Bibin A Chundatt , also did some tests locally. Sunil G mentioned the root cause. Currently, we only have nodeLabelExpression assigned to off-switch request, and nodeLabelExpression of node-local/rack-local request is always empty. So SchedulerUtils.checkResourceRequestMatchingNodePartition will be always failed for node-local/rack-local requests. And now, computation of off-switch delay has some issues, wait time is linear correlated to #requested-container / #hosts. (Will file ticket to fix that problem.) So in worst cases, scheduler will wait for more than several minutes to get the first off-switch container. In this JIRA, I think we need to fix the first problem, we don't allow to set nodeLabelExpression for node-local/rack-local request after YARN-2694 , but I think we need to make nodeLabelExpression of node-local/rack-local consistent with off-switch requests. Back to the code, I think what need to fix is, in AppSchedulingInfo#updateResourceRequests. When (request.resourceName == ANY && request.node-label-expression updated) we need to update all resource requests of this priority When (request.resourceName != ANY) we need to set request.node-label-expression equals to ANY-resource-request.node-label-expression of the same priority Thoughts?
          Hide
          sunilg Sunil G added a comment -

          HI Wangda Tan

          I have a doubt here. node-label expression is set in ANY by AM. Any reason why its not updated for node-local and rack-local there itself. Could you pls help to clarify.

          Show
          sunilg Sunil G added a comment - HI Wangda Tan I have a doubt here. node-label expression is set in ANY by AM. Any reason why its not updated for node-local and rack-local there itself. Could you pls help to clarify.
          Hide
          leftnoteasy Wangda Tan added a comment -

          We force client doesn't set node-label expression (in YARN-2694) because we don't want client set different node-label-expression for different resourceName in a same priority (for priority=2, "rack-1"'s node-label-expression="x", but ""'s node-label-expression="y"). Remember we count pendingResource by using "" of each priority.

          But we can normalize node-label-expression once they sent to scheduler.

          Make sense?

          Show
          leftnoteasy Wangda Tan added a comment - We force client doesn't set node-label expression (in YARN-2694 ) because we don't want client set different node-label-expression for different resourceName in a same priority (for priority=2, "rack-1"'s node-label-expression="x", but " "'s node-label-expression="y"). Remember we count pendingResource by using " " of each priority. But we can normalize node-label-expression once they sent to scheduler. Make sense?
          Hide
          sunilg Sunil G added a comment -

          Yes Wangda Tan
          Thank you clarifying the same. This makes sense.

          Show
          sunilg Sunil G added a comment - Yes Wangda Tan Thank you clarifying the same. This makes sense.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching first patch for review

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching first patch for review
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          hi Bibin A Chundatt,
          Thanks for updating with a patch. Just took a high level look at it. Seems like we can optimize a bit here as node labels might not be always set.
          Currently you are always looping through twice, instead first while checking for any Requests and we can populate anyPrioritymap map with additional check

          (null != anyResourceRequest.getNodeLabelExpression())
                      && (!anyResourceRequest.getNodeLabelExpression().equals(
                          RMNodeLabelsManager.NO_LABEL))
          

          after this loop if the map contains any element then we can loop for updating node & rack local request with NodeLabelExpression
          Also HashMap<Priority, ResourceRequest> anyPrioritymap => Map<Priority, String> priorityToNodeLabelMapping

          Show
          Naganarasimha Naganarasimha G R added a comment - hi Bibin A Chundatt , Thanks for updating with a patch. Just took a high level look at it. Seems like we can optimize a bit here as node labels might not be always set. Currently you are always looping through twice, instead first while checking for any Requests and we can populate anyPrioritymap map with additional check ( null != anyResourceRequest.getNodeLabelExpression()) && (!anyResourceRequest.getNodeLabelExpression().equals( RMNodeLabelsManager.NO_LABEL)) after this loop if the map contains any element then we can loop for updating node & rack local request with NodeLabelExpression Also HashMap<Priority, ResourceRequest> anyPrioritymap => Map<Priority, String> priorityToNodeLabelMapping
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Naganarasimha G R
          Thnks for the comment missed it. Have updated patch as per comments.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Naganarasimha G R Thnks for the comment missed it. Have updated patch as per comments.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching updated patch with testcases

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching updated patch with testcases
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 56s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 58s There were no new javac warning messages.
          +1 javadoc 10m 18s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 50s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse.
          +1 findbugs 1m 30s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 54m 58s Tests passed in hadoop-yarn-server-resourcemanager.
              95m 2s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12755709/0003-YARN-4140.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6955771
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9120/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9120/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9120/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 56s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 58s There were no new javac warning messages. +1 javadoc 10m 18s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 50s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse. +1 findbugs 1m 30s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 54m 58s Tests passed in hadoop-yarn-server-resourcemanager.     95m 2s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755709/0003-YARN-4140.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6955771 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9120/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9120/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9120/console This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Bibin A Chundatt,
          Few comments:

          • It's no need to maintain an anyPrioritymap, getResourceRequest(Priority, resourceName) should enough.

          Current logic seems not correct to me, it should cover following cases:

          • Newly-added-non-ANY request should have same labelExpression of ANY request. (done in your patch already)
          • If ANY request newly added OR labelExpression changed of ANY request, you should update ALL resource requests' labelExpression.

          And if you agree about above logic, tests should cover above cases as well.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Bibin A Chundatt , Few comments: It's no need to maintain an anyPrioritymap , getResourceRequest(Priority, resourceName) should enough. Current logic seems not correct to me, it should cover following cases: Newly-added-non-ANY request should have same labelExpression of ANY request. (done in your patch already) If ANY request newly added OR labelExpression changed of ANY request, you should update ALL resource requests' labelExpression. And if you agree about above logic, tests should cover above cases as well.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Wangda Tan

          Thnks for the comments

          Have a doubt regarding the below point

          If ANY request newly added OR labelExpression changed of ANY request, you should update ALL resource requests' labelExpression.

          This case will happen only when at run time we are changing label for an application rt??
          Is it possible to change label of an application at runtime??

          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Wangda Tan Thnks for the comments Have a doubt regarding the below point If ANY request newly added OR labelExpression changed of ANY request, you should update ALL resource requests' labelExpression. This case will happen only when at run time we are changing label for an application rt?? Is it possible to change label of an application at runtime??
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching patch as per the comments

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching patch as per the comments
          Hide
          leftnoteasy Wangda Tan added a comment -

          Bibin A Chundatt,
          Thanks for update, I think the structure could be optimized:

          In my mind, it could be:

          for (request : requests):
          	if (request.name == ANY):
          		// Original logics ............
          
          		ResourceRequest oldAny = getRequest(request.priority, ANY);
          		if oldAny == null || oldAny.getLabel != request.getLabel:
          			// Update all requests in the priority
          			for (r: getRequests(request.priority)):
          				r.label = request.label
          	else:
          		// Original logics ............
          
          		ResourceRequest any = getRequest(request.priority, ANY);
          		if (any != null):
          			request.label = any.label
          
          Show
          leftnoteasy Wangda Tan added a comment - Bibin A Chundatt , Thanks for update, I think the structure could be optimized: In my mind, it could be: for (request : requests): if (request.name == ANY): // Original logics ............ ResourceRequest oldAny = getRequest(request.priority, ANY); if oldAny == null || oldAny.getLabel != request.getLabel: // Update all requests in the priority for (r: getRequests(request.priority)): r.label = request.label else : // Original logics ............ ResourceRequest any = getRequest(request.priority, ANY); if (any != null ): request.label = any.label
          Hide
          leftnoteasy Wangda Tan added a comment -

          To explain: above logic can avoid two more cases:

          • Previous node/rack request has label="x", but now we update its any request's label="" (or vice versa)
          • Previous we don't have ANY request, this may not happen, but we can protect this corner case.
          Show
          leftnoteasy Wangda Tan added a comment - To explain: above logic can avoid two more cases: Previous node/rack request has label="x", but now we update its any request's label="" (or vice versa) Previous we don't have ANY request, this may not happen, but we can protect this corner case.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Wangda Tan

          The request order ANY will not be always the first one rt? So i Think in first updateRequest will miss update of elements prior to ANY. Thts the reason i added 2 loops.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Wangda Tan The request order ANY will not be always the first one rt? So i Think in first updateRequest will miss update of elements prior to ANY. Thts the reason i added 2 loops.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Bibin A Chundatt,

          The request order ANY will not be always the first one rt?

          It won't be always the first one.

          The pseudo code should be able to handle request.label correctly whether ANY requests come first or not.

          Show
          leftnoteasy Wangda Tan added a comment - Bibin A Chundatt , The request order ANY will not be always the first one rt? It won't be always the first one. The pseudo code should be able to handle request.label correctly whether ANY requests come first or not.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Thanks Wangda Tan for review and comments.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Thanks Wangda Tan for review and comments.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 10s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 1s There were no new javac warning messages.
          +1 javadoc 10m 27s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 53s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 yarn tests 55m 41s Tests failed in hadoop-yarn-server-resourcemanager.
              96m 19s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
            hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
            hadoop.yarn.server.resourcemanager.TestClientRMService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12757131/0005-YARN-4140.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3f82f58
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9196/artifact/patchprocess/whitespace.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9196/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9196/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9196/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 10s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 1s There were no new javac warning messages. +1 javadoc 10m 27s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 53s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 55m 41s Tests failed in hadoop-yarn-server-resourcemanager.     96m 19s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation   hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler   hadoop.yarn.server.resourcemanager.TestClientRMService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12757131/0005-YARN-4140.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3f82f58 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9196/artifact/patchprocess/whitespace.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9196/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9196/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9196/console This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching patch after update

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching patch after update
          Hide
          leftnoteasy Wangda Tan added a comment -

          Bibin A Chundatt, could you take a look at test failures? They look like related to changes of this patch.

          Show
          leftnoteasy Wangda Tan added a comment - Bibin A Chundatt , could you take a look at test failures? They look like related to changes of this patch.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Have attached 0006-YARN-4140 after testcase fix scheduler testcase are related .Others are failing with out patch also.Will file a jira for that.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Have attached 0006- YARN-4140 after testcase fix scheduler testcase are related .Others are failing with out patch also.Will file a jira for that.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Wangda Tan

          Details of testcase failures

          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
          hadoop.yarn.server.resourcemanager.TestClientRMService 
          

          These two are not related to patch uploaded

          hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
          hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
          

          Fixed

          Jenkins not triggered due to git pull failure. Uploading new patch to trigger jenkins again

          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Wangda Tan Details of testcase failures hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation hadoop.yarn.server.resourcemanager.TestClientRMService These two are not related to patch uploaded hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler Fixed Jenkins not triggered due to git pull failure . Uploading new patch to trigger jenkins again
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 40s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 47s There were no new javac warning messages.
          +1 javadoc 9m 58s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 51s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 yarn tests 57m 24s Tests failed in hadoop-yarn-server-resourcemanager.
              96m 39s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
            hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761269/0007-YARN-4140.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / e3ace31
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9218/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9218/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9218/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 47s There were no new javac warning messages. +1 javadoc 9m 58s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 51s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 57m 24s Tests failed in hadoop-yarn-server-resourcemanager.     96m 39s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation   hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761269/0007-YARN-4140.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e3ace31 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9218/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9218/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9218/console This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Failures are not due to current patch.

          hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
          hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter 
          

          Reason class not found and TestNodeLabelContainerAllocation testcases failed on trunk even with out patch.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Failures are not due to current patch. hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter Reason class not found and TestNodeLabelContainerAllocation testcases failed on trunk even with out patch.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Will recheck {{TestNodeLabelContainerAllocation }} failures

          Show
          bibinchundatt Bibin A Chundatt added a comment - Will recheck {{TestNodeLabelContainerAllocation }} failures
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          They are related will check how to update testcases for that.

          Show
          bibinchundatt Bibin A Chundatt added a comment - They are related will check how to update testcases for that.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Wangda Tan

          Could you please review the patch attached.
          When labels updated for any then pending resource usage for queue and app also need updated rt?. Have changed based on that .

          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Wangda Tan Could you please review the patch attached. When labels updated for any then pending resource usage for queue and app also need updated rt?. Have changed based on that .
          Hide
          sunilg Sunil G added a comment -

          Hi Bibin A Chundatt
          Thanks for updating patch.

          Some minor nits:
          1. incPendingResourcesForLabel and decPendingResourceForLabel need not have to take ResourceRequest as argument. Only label expression is to be passed along with resource.
          2. In below code

                } else {
                  ResourceRequest anyRequest = getResourceRequest(priority,
                      ResourceRequest.ANY);
                  if (anyRequest != null) {
                    request.setNodeLabelExpression(anyRequest.getNodeLabelExpression());
                  }
               }
          

          for any other resource requests, label expression is set as from anyRequest. One of point here

          • If user is not specified any label expression, then also we forcefully set anyRequest.getNodeLabelExpression() in all requests. It can be null too. Such cases can be invalidated.

          3. In testResourceRequestUpdateNodePartitions, before sending second changed AM resource request, could you also add few more NODE_LOCAL or RACK_LOCAL (some priority to ANY, and some after ANY). This can help in hitting some more areas in code.

          Show
          sunilg Sunil G added a comment - Hi Bibin A Chundatt Thanks for updating patch. Some minor nits: 1. incPendingResourcesForLabel and decPendingResourceForLabel need not have to take ResourceRequest as argument. Only label expression is to be passed along with resource. 2. In below code } else { ResourceRequest anyRequest = getResourceRequest(priority, ResourceRequest.ANY); if (anyRequest != null ) { request.setNodeLabelExpression(anyRequest.getNodeLabelExpression()); } } for any other resource requests, label expression is set as from anyRequest. One of point here If user is not specified any label expression, then also we forcefully set anyRequest.getNodeLabelExpression() in all requests. It can be null too. Such cases can be invalidated. 3. In testResourceRequestUpdateNodePartitions, before sending second changed AM resource request, could you also add few more NODE_LOCAL or RACK_LOCAL (some priority to ANY, and some after ANY). This can help in hitting some more areas in code.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 41s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 59s There were no new javac warning messages.
          +1 javadoc 10m 16s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 49s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 yarn tests 57m 33s Tests failed in hadoop-yarn-server-resourcemanager.
              97m 18s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
            hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
          Timed out tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761638/0008-YARN-4140.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 57003fa
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9232/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9232/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9232/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 41s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 59s There were no new javac warning messages. +1 javadoc 10m 16s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 49s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 57m 33s Tests failed in hadoop-yarn-server-resourcemanager.     97m 18s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler   hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler Timed out tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761638/0008-YARN-4140.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 57003fa hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9232/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9232/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9232/console This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Sunil G

          Thnks for review and comments.
          Have updated restcases and patch as per comments

          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Sunil G Thnks for review and comments. Have updated restcases and patch as per comments
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 44s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 7s There were no new javac warning messages.
          +1 javadoc 10m 10s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 49s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 29s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 yarn tests 54m 23s Tests failed in hadoop-yarn-server-resourcemanager.
              96m 15s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761696/0009-YARN-4140.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / cc2b473
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9233/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9233/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9233/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 44s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 7s There were no new javac warning messages. +1 javadoc 10m 10s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 49s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 54m 23s Tests failed in hadoop-yarn-server-resourcemanager.     96m 15s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761696/0009-YARN-4140.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / cc2b473 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9233/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9233/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9233/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          thanks for the patch Bibin A Chundatt, and for the approach by Tan, Wangda,
          But i had few queries :

          1. Previous we don't have ANY request, this may not happen, but we can protect this corner case.

            Can this scenario be possible ? IIRC, MR its not possible but may be possible in other apps ?
            If so is it req to handle incPendingResourcesForLabel and decPendingResourceForLabel for such requests currently we are doing for only previousAnyRequest only .

          2. I think the modifications for updates would not be required when recoverPreemptedRequest is true right ?

          Small nit, not exactly related to the patch, method parameter requests could be renamed as updatedRequests and the local variable in the for loop as updatedRequest, so that its more readable as currently a member with the same name requests

          Show
          Naganarasimha Naganarasimha G R added a comment - thanks for the patch Bibin A Chundatt , and for the approach by Tan, Wangda , But i had few queries : Previous we don't have ANY request, this may not happen, but we can protect this corner case. Can this scenario be possible ? IIRC, MR its not possible but may be possible in other apps ? If so is it req to handle incPendingResourcesForLabel and decPendingResourceForLabel for such requests currently we are doing for only previousAnyRequest only . I think the modifications for updates would not be required when recoverPreemptedRequest is true right ? Small nit, not exactly related to the patch, method parameter requests could be renamed as updatedRequests and the local variable in the for loop as updatedRequest , so that its more readable as currently a member with the same name requests
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Synced offline with NGarla_Unused

          If so is it req to handle incPendingResourcesForLabel and decPendingResourceForLabel for such requests currently we are doing for only previousAnyRequest only .

          Currrently the increment and decrements is happening only for Any request so updates not required .

          I think the modifications for updates would not be required when recoverPreemptedRequest is true right ?

          Not mandatory change to be done.

          currently a member with the same name requests

          As part of YARN-4157 the same is getting changed

          Show
          bibinchundatt Bibin A Chundatt added a comment - Synced offline with NGarla_Unused If so is it req to handle incPendingResourcesForLabel and decPendingResourceForLabel for such requests currently we are doing for only previousAnyRequest only . Currrently the increment and decrements is happening only for Any request so updates not required . I think the modifications for updates would not be required when recoverPreemptedRequest is true right ? Not mandatory change to be done. currently a member with the same name requests As part of YARN-4157 the same is getting changed
          Hide
          leftnoteasy Wangda Tan added a comment -

          Can this scenario be possible ? IIRC, MR its not possible but may be possible in other apps ?

          I remember we have previously solved some corner cases like this, I'm not 100% sure, but I assume this could happen.

          Currrently the increment and decrements is happening only for Any request so updates not required .

          Agree

          Bibin A Chundatt, could you update the patch to latest trunk?

          Show
          leftnoteasy Wangda Tan added a comment - Can this scenario be possible ? IIRC, MR its not possible but may be possible in other apps ? I remember we have previously solved some corner cases like this, I'm not 100% sure, but I assume this could happen. Currrently the increment and decrements is happening only for Any request so updates not required . Agree Bibin A Chundatt , could you update the patch to latest trunk?
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Uploading patch after rebasing based on latest trunk

          Show
          bibinchundatt Bibin A Chundatt added a comment - Uploading patch after rebasing based on latest trunk
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Bibin A Chundatt & Tan, Wangda,
          Thanks for clarifications,

          I remember we have previously solved some corner cases like this, I'm not 100% sure, but I assume this could happen.

          i thought if ((null == previousAnyRequest) was added only for this case and inc and dec are not handled but as Bibin A Chundatt clarified its not only for that and the corner case doesn't completely handled as increment and decrements is happening only for Any request.
          Latest patch lgtm.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Bibin A Chundatt & Tan, Wangda , Thanks for clarifications, I remember we have previously solved some corner cases like this, I'm not 100% sure, but I assume this could happen. i thought if ((null == previousAnyRequest) was added only for this case and inc and dec are not handled but as Bibin A Chundatt clarified its not only for that and the corner case doesn't completely handled as increment and decrements is happening only for Any request. Latest patch lgtm.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 11s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 6s There were no new javac warning messages.
          +1 javadoc 10m 18s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 51s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 46s mvn install still works.
          +1 eclipse:eclipse 0m 40s The patch built with eclipse:eclipse.
          +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 66m 55s Tests passed in hadoop-yarn-server-resourcemanager.
              107m 57s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762301/0010-YARN-4140.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d1b9b85
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9263/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9263/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9263/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 11s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 6s There were no new javac warning messages. +1 javadoc 10m 18s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 51s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 46s mvn install still works. +1 eclipse:eclipse 0m 40s The patch built with eclipse:eclipse. +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 66m 55s Tests passed in hadoop-yarn-server-resourcemanager.     107m 57s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762301/0010-YARN-4140.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d1b9b85 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9263/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9263/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9263/console This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          Hi Bibin A Chundatt
          One more minor nit:

          private boolean isRequestLabelChanged(ResourceRequest currentRequest, ResourceRequest previousRequest)

          I think Its better we can change argument names here to requestOne, requestTwo as we are already using it in both contexts.

          Show
          sunilg Sunil G added a comment - Hi Bibin A Chundatt One more minor nit: private boolean isRequestLabelChanged(ResourceRequest currentRequest, ResourceRequest previousRequest) I think Its better we can change argument names here to requestOne , requestTwo as we are already using it in both contexts.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Sunil G

          Updating params as per comment

          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Sunil G Updating params as per comment
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 9s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 55s There were no new javac warning messages.
          +1 javadoc 10m 4s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 48s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 yarn tests 55m 43s Tests failed in hadoop-yarn-server-resourcemanager.
              95m 36s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
            hadoop.yarn.server.resourcemanager.TestClientRMService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762401/0011-YARN-4140.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 83e65c5
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9266/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9266/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9266/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 9s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 55s There were no new javac warning messages. +1 javadoc 10m 4s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 48s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 55m 43s Tests failed in hadoop-yarn-server-resourcemanager.     95m 36s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService   hadoop.yarn.server.resourcemanager.TestClientRMService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762401/0011-YARN-4140.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 83e65c5 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9266/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9266/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9266/console This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Failures are not related to his patch

          Show
          bibinchundatt Bibin A Chundatt added a comment - Failures are not related to his patch
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 16m 2s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 12s There were no new javac warning messages.
          +1 javadoc 10m 29s There were no new javadoc warning messages.
          +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 42s The applied patch generated 39 new checkstyle issues (total was 0, now 39).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
          +1 findbugs 1m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 yarn tests 61m 49s Tests failed in hadoop-yarn-server-resourcemanager.
              101m 29s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12764031/0012-YARN-4140.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 892ade6
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9286/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9286/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9286/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9286/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 2s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 12s There were no new javac warning messages. +1 javadoc 10m 29s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 42s The applied patch generated 39 new checkstyle issues (total was 0, now 39). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 1m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 61m 49s Tests failed in hadoop-yarn-server-resourcemanager.     101m 29s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12764031/0012-YARN-4140.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 892ade6 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9286/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9286/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9286/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9286/console This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Checkstyle issues are not from modified lines of code and Testcase failure are unrelated

          Show
          bibinchundatt Bibin A Chundatt added a comment - Checkstyle issues are not from modified lines of code and Testcase failure are unrelated
          Hide
          leftnoteasy Wangda Tan added a comment -

          1) This part is not necessary:

                  // We should update queue and appUsage based on new label in case of ANY
                  // request.When ANY request label is changed, we should update pending
                  // resource queue,appUsage based on latest ANY request label.
                  if ((null != previousAnyRequest)
                      && isRequestLabelChanged(request, previousAnyRequest)) {
                    Resource modResource =
                        Resources.multiply(previousAnyRequest.getCapability(),
                            previousAnyRequest.getNumContainers());
                    decPendingResourceForLabel(
                        previousAnyRequest.getNodeLabelExpression(), modResource);
                    incPendingResourcesForLabel(request.getNodeLabelExpression(),
                        modResource);
                  }
          

          We did updates at

              if (updatePendingResources) {
                 ...
              }
          

          And it calculated wrong value for increased resource.

          2) Naming suggestions:

                  // When there is change in ANY request label expression, we should
                  // update label for all resource requests already added of same
                  // priority as ANY resource request.
                  if ((null == previousAnyRequest)
                      || isRequestLabelChanged(previousAnyRequest, request)) {
                    Map<String, ResourceRequest> preResourceRequests =
                        getResourceRequests(priority);
                    if (preResourceRequests != null) {
                      for (ResourceRequest oldRequest : preResourceRequests.values()) {
                        oldRequest.setNodeLabelExpression(request
                            .getNodeLabelExpression());
                      }
                    }
                  }
          
          • preResourceRequests -> resourceRequests, they're not "pre"
          • oldRequest -> r or something, they're not "old"

          3) Following part:

          else {
                  ResourceRequest anyRequest =
                      getResourceRequest(priority, ResourceRequest.ANY);
                  if (anyRequest != null && isRequestLabelChanged(anyRequest, request)) {
                    request.setNodeLabelExpression(anyRequest.getNodeLabelExpression());
                  }
          

          no need to do isRequestLabelChanged check, setNodeLabelExpression when any != null

          4) isRequestLabelChanged may not needed, or you don't need to check nodeLabelExp == null, org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils#normalizeNodeLabelExpressionInRequest make sure it won't be null.

          Show
          leftnoteasy Wangda Tan added a comment - 1) This part is not necessary: // We should update queue and appUsage based on new label in case of ANY // request.When ANY request label is changed, we should update pending // resource queue,appUsage based on latest ANY request label. if (( null != previousAnyRequest) && isRequestLabelChanged(request, previousAnyRequest)) { Resource modResource = Resources.multiply(previousAnyRequest.getCapability(), previousAnyRequest.getNumContainers()); decPendingResourceForLabel( previousAnyRequest.getNodeLabelExpression(), modResource); incPendingResourcesForLabel(request.getNodeLabelExpression(), modResource); } We did updates at if (updatePendingResources) { ... } And it calculated wrong value for increased resource. 2) Naming suggestions: // When there is change in ANY request label expression, we should // update label for all resource requests already added of same // priority as ANY resource request. if (( null == previousAnyRequest) || isRequestLabelChanged(previousAnyRequest, request)) { Map< String , ResourceRequest> preResourceRequests = getResourceRequests(priority); if (preResourceRequests != null ) { for (ResourceRequest oldRequest : preResourceRequests.values()) { oldRequest.setNodeLabelExpression(request .getNodeLabelExpression()); } } } preResourceRequests -> resourceRequests, they're not "pre" oldRequest -> r or something, they're not "old" 3) Following part: else { ResourceRequest anyRequest = getResourceRequest(priority, ResourceRequest.ANY); if (anyRequest != null && isRequestLabelChanged(anyRequest, request)) { request.setNodeLabelExpression(anyRequest.getNodeLabelExpression()); } no need to do isRequestLabelChanged check, setNodeLabelExpression when any != null 4) isRequestLabelChanged may not needed, or you don't need to check nodeLabelExp == null, org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils#normalizeNodeLabelExpressionInRequest make sure it won't be null.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Wangda Tan

          Thanks for the review and comments.
          Have updated patched based on comments.

          1. Comment 1: Got confused with resource request asks.Only the last Any request label need to be stored for decreasing usages in
             if (updatePendingResources) {
                   ...
                }
            
          2. Comment 2: Udpated the variables name as per comments.
          3. Comment 3: Only when change happens wanted to change the request thats the reason added the same.
          4. Comment 4: Faced Fair and Fifo related testcase failures when request is label was null.Currently the same is not happening.
          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Wangda Tan Thanks for the review and comments. Have updated patched based on comments. Comment 1: Got confused with resource request asks.Only the last Any request label need to be stored for decreasing usages in if (updatePendingResources) { ... } Comment 2: Udpated the variables name as per comments. Comment 3: Only when change happens wanted to change the request thats the reason added the same. Comment 4: Faced Fair and Fifo related testcase failures when request is label was null.Currently the same is not happening.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 36s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 14s There were no new javac warning messages.
          +1 javadoc 10m 8s There were no new javadoc warning messages.
          -1 release audit 0m 16s The applied patch generated 1 release audit warnings.
          +1 checkstyle 0m 50s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 59m 59s Tests passed in hadoop-yarn-server-resourcemanager.
              100m 38s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12764536/0013-YARN-4140.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 5db371f
          Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9317/artifact/patchprocess/patchReleaseAuditProblems.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9317/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9317/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9317/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 14s There were no new javac warning messages. +1 javadoc 10m 8s There were no new javadoc warning messages. -1 release audit 0m 16s The applied patch generated 1 release audit warnings. +1 checkstyle 0m 50s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 59m 59s Tests passed in hadoop-yarn-server-resourcemanager.     100m 38s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12764536/0013-YARN-4140.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5db371f Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9317/artifact/patchprocess/patchReleaseAuditProblems.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9317/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9317/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9317/console This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Hi Bibin A Chundatt,
          Thanks for update, several minor comments:
          1)
          String lastAnyRequestLabel = null; is not necessary.

          2)
          Changes after

                  // update queue:
                  Resource increasedResource =
                      Resources.multiply(request.getCapability(),
                          request.getNumContainers());
          

          Could be removed

          3) isRequestLabelChanged still checks null of expression, could be removed as well. Fair/FIFO tests could be failed because of this, this is caused by they are using mockito so requests aren't processed by SchedulerUtils.normalize... I suggest fix test instead of putting the logic to main code.

          Show
          leftnoteasy Wangda Tan added a comment - Hi Bibin A Chundatt , Thanks for update, several minor comments: 1) String lastAnyRequestLabel = null; is not necessary. 2) Changes after // update queue: Resource increasedResource = Resources.multiply(request.getCapability(), request.getNumContainers()); Could be removed 3) isRequestLabelChanged still checks null of expression, could be removed as well. Fair/FIFO tests could be failed because of this, this is caused by they are using mockito so requests aren't processed by SchedulerUtils.normalize... I suggest fix test instead of putting the logic to main code.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Uploaded new patch for the same.

          1. String lastAnyRequestLabel = null; is not necessary. - Added check not update label for ANY since it gets handled already as part of code available and lastRequest label will be correct for queue and appUsage.
          2. Only formating is done now.
          3. Handled FIFO and Fair testcase correction
          Show
          bibinchundatt Bibin A Chundatt added a comment - Uploaded new patch for the same. String lastAnyRequestLabel = null; is not necessary. - Added check not update label for ANY since it gets handled already as part of code available and lastRequest label will be correct for queue and appUsage. Only formating is done now. Handled FIFO and Fair testcase correction
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 33s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 51s There were no new javac warning messages.
          +1 javadoc 10m 2s There were no new javadoc warning messages.
          -1 release audit 0m 16s The applied patch generated 1 release audit warnings.
          +1 checkstyle 1m 12s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 26s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 60m 14s Tests passed in hadoop-yarn-server-resourcemanager.
              102m 14s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12764796/0014-YARN-4140.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 439f43a
          Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9331/artifact/patchprocess/patchReleaseAuditProblems.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9331/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9331/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9331/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9331/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 33s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 51s There were no new javac warning messages. +1 javadoc 10m 2s There were no new javadoc warning messages. -1 release audit 0m 16s The applied patch generated 1 release audit warnings. +1 checkstyle 1m 12s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 26s Tests passed in hadoop-yarn-server-common. +1 yarn tests 60m 14s Tests passed in hadoop-yarn-server-resourcemanager.     102m 14s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12764796/0014-YARN-4140.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 439f43a Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9331/artifact/patchprocess/patchReleaseAuditProblems.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9331/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9331/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9331/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9331/console This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks for update, Bibin A Chundatt, patch looks good, pending Jenkins.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks for update, Bibin A Chundatt , patch looks good, pending Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 20m 35s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 9m 0s There were no new javac warning messages.
          +1 javadoc 11m 40s There were no new javadoc warning messages.
          -1 release audit 0m 21s The applied patch generated 1 release audit warnings.
          +1 checkstyle 1m 23s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 46s mvn install still works.
          +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
          +1 findbugs 2m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 29s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 63m 31s Tests passed in hadoop-yarn-server-resourcemanager.
              112m 23s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12764796/0014-YARN-4140.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 8d22622
          Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9385/artifact/patchprocess/patchReleaseAuditProblems.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9385/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9385/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9385/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9385/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 35s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 9m 0s There were no new javac warning messages. +1 javadoc 11m 40s There were no new javadoc warning messages. -1 release audit 0m 21s The applied patch generated 1 release audit warnings. +1 checkstyle 1m 23s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 46s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. +1 findbugs 2m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 29s Tests passed in hadoop-yarn-server-common. +1 yarn tests 63m 31s Tests passed in hadoop-yarn-server-resourcemanager.     112m 23s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12764796/0014-YARN-4140.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8d22622 Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9385/artifact/patchprocess/patchReleaseAuditProblems.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9385/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9385/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9385/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9385/console This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Wangda Tan

          Thanks for looking into it.Release audit warning not related to current patch

          /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/util/tree.h
          Lines that start with ????? in the release audit  report indicate files that do not have an Apache license header.
          
          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Wangda Tan Thanks for looking into it.Release audit warning not related to current patch /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/util/tree.h Lines that start with ????? in the release audit report indicate files that do not have an Apache license header.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Committed to trunk/branch-2, thanks Bibin A Chundatt and review from Naganarasimha G R and Sunil G!

          Show
          leftnoteasy Wangda Tan added a comment - Committed to trunk/branch-2, thanks Bibin A Chundatt and review from Naganarasimha G R and Sunil G !
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #8604 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8604/)
          YARN-4140. RM container allocation delayed incase of app submitted to (wangda: rev def374e666ed0c1d665aeb1b7307e09769448138)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #8604 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8604/ ) YARN-4140 . RM container allocation delayed incase of app submitted to (wangda: rev def374e666ed0c1d665aeb1b7307e09769448138) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1244 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1244/)
          YARN-4140. RM container allocation delayed incase of app submitted to (wangda: rev def374e666ed0c1d665aeb1b7307e09769448138)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1244 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1244/ ) YARN-4140 . RM container allocation delayed incase of app submitted to (wangda: rev def374e666ed0c1d665aeb1b7307e09769448138) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #518 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/518/)
          YARN-4140. RM container allocation delayed incase of app submitted to (wangda: rev def374e666ed0c1d665aeb1b7307e09769448138)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #518 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/518/ ) YARN-4140 . RM container allocation delayed incase of app submitted to (wangda: rev def374e666ed0c1d665aeb1b7307e09769448138) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #507 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/507/)
          YARN-4140. RM container allocation delayed incase of app submitted to (wangda: rev def374e666ed0c1d665aeb1b7307e09769448138)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #507 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/507/ ) YARN-4140 . RM container allocation delayed incase of app submitted to (wangda: rev def374e666ed0c1d665aeb1b7307e09769448138) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java hadoop-yarn-project/CHANGES.txt
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Thank you Wangda Tan for review and committing

          Show
          bibinchundatt Bibin A Chundatt added a comment - Thank you Wangda Tan for review and committing
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2452 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2452/)
          YARN-4140. RM container allocation delayed incase of app submitted to (wangda: rev def374e666ed0c1d665aeb1b7307e09769448138)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2452 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2452/ ) YARN-4140 . RM container allocation delayed incase of app submitted to (wangda: rev def374e666ed0c1d665aeb1b7307e09769448138) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #479 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/479/)
          YARN-4140. RM container allocation delayed incase of app submitted to (wangda: rev def374e666ed0c1d665aeb1b7307e09769448138)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #479 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/479/ ) YARN-4140 . RM container allocation delayed incase of app submitted to (wangda: rev def374e666ed0c1d665aeb1b7307e09769448138) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2417 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2417/)
          YARN-4140. RM container allocation delayed incase of app submitted to (wangda: rev def374e666ed0c1d665aeb1b7307e09769448138)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2417 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2417/ ) YARN-4140 . RM container allocation delayed incase of app submitted to (wangda: rev def374e666ed0c1d665aeb1b7307e09769448138) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java hadoop-yarn-project/CHANGES.txt
          Hide
          leftnoteasy Wangda Tan added a comment -

          I think we may need to consider to put this to next 2.7 release, this solved allocation issue when using MR (or other locality required application under node label). Thoughts? Bibin A Chundatt/Sunil G/Naganarasimha G R.

          Show
          leftnoteasy Wangda Tan added a comment - I think we may need to consider to put this to next 2.7 release, this solved allocation issue when using MR (or other locality required application under node label). Thoughts? Bibin A Chundatt / Sunil G / Naganarasimha G R .
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Yes Tan, Wangda, we need to atleast selectively pick few jira's from the list i shared in forum, if not usability will be an issue for the guys who are using based on 2.7 versions. i think even Webui indicating the partition information and the REST are all basic requirements to use NodeLabels.

          Show
          Naganarasimha Naganarasimha G R added a comment - Yes Tan, Wangda , we need to atleast selectively pick few jira's from the list i shared in forum, if not usability will be an issue for the guys who are using based on 2.7 versions. i think even Webui indicating the partition information and the REST are all basic requirements to use NodeLabels.
          Hide
          sunilg Sunil G added a comment -

          This ticket is good candidate for 2.7 line as we will be running in to this problem mostly with MR cases. +1.

          As for the list, I feel most of them are fine. But we can try avoid some big patches from that as it may not be tested thoroughly. WebUI and REST related tickets wil be very useful for 2.7 line of release as it gives more details and clarity for labels (easy to understand allocation details and for debugging)

          Show
          sunilg Sunil G added a comment - This ticket is good candidate for 2.7 line as we will be running in to this problem mostly with MR cases. +1. As for the list, I feel most of them are fine. But we can try avoid some big patches from that as it may not be tested thoroughly. WebUI and REST related tickets wil be very useful for 2.7 line of release as it gives more details and clarity for labels (easy to understand allocation details and for debugging)
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          +1 for merging in 2.7 .

          Show
          bibinchundatt Bibin A Chundatt added a comment - +1 for merging in 2.7 .
          Hide
          leftnoteasy Wangda Tan added a comment -

          Naganarasimha G R/Sunil G, I agree we should try best to stabilize node label for 2.7.x releases. And I understand webUI/REST API are all very important.

          My biggest concern is many of these REST API fixes are feature/improvements instead of bug fixes, and they're all non-trivial patches. I would prefer to commit only critical and/or safe bug fixes to branch-2.7. And we're planning 2.8 now, it has many improvements/stabilizations to node label features, I think we should suggest people to try 2.8 if they want to use node label in production.

          Show
          leftnoteasy Wangda Tan added a comment - Naganarasimha G R / Sunil G , I agree we should try best to stabilize node label for 2.7.x releases. And I understand webUI/REST API are all very important. My biggest concern is many of these REST API fixes are feature/improvements instead of bug fixes, and they're all non-trivial patches. I would prefer to commit only critical and/or safe bug fixes to branch-2.7. And we're planning 2.8 now, it has many improvements/stabilizations to node label features, I think we should suggest people to try 2.8 if they want to use node label in production.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Yes Tan, Wangda you are right, if 2.8 is in near future then there is no point in pushing these issues to 2.7.x release. But all our fixes/improvements makes the NodeLabel more usable hence was trying to push at least the important ones !

          Show
          Naganarasimha Naganarasimha G R added a comment - Yes Tan, Wangda you are right, if 2.8 is in near future then there is no point in pushing these issues to 2.7.x release. But all our fixes/improvements makes the NodeLabel more usable hence was trying to push at least the important ones !

            People

            • Assignee:
              bibinchundatt Bibin A Chundatt
              Reporter:
              bibinchundatt Bibin A Chundatt
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development