Details

      Description

      Target of this JIRA:

      • Definition of interfaces / objects which will be used by global scheduling, this will be shared by different schedulers.
      • Modify CapacityScheduler to use it.
      1. YARN-5716.001.patch
        396 kB
        Wangda Tan
      2. YARN-5716.0012.patch
        434 kB
        Wangda Tan
      3. YARN-5716.002.patch
        418 kB
        Wangda Tan
      4. YARN-5716.003.patch
        421 kB
        Wangda Tan
      5. YARN-5716.004.patch
        422 kB
        Wangda Tan
      6. YARN-5716.005.patch
        421 kB
        Wangda Tan
      7. YARN-5716.006.patch
        420 kB
        Wangda Tan
      8. YARN-5716.007.patch
        425 kB
        Wangda Tan
      9. YARN-5716.008.patch
        424 kB
        Wangda Tan
      10. YARN-5716.009.patch
        424 kB
        Wangda Tan
      11. YARN-5716.010.patch
        458 kB
        Wangda Tan
      12. YARN-5716.011.patch
        434 kB
        Wangda Tan
      13. YARN-5716.013.patch
        433 kB
        Wangda Tan

        Activity

        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached ver.1 patch for review, it is fully functioned, has all changes needed in my mind right now. Will clean up more in the next patch for things like unnecessary comments, indent, etc.

        Show
        leftnoteasy Wangda Tan added a comment - Attached ver.1 patch for review, it is fully functioned, has all changes needed in my mind right now. Will clean up more in the next patch for things like unnecessary comments, indent, etc.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 12 new or modified test files.
        0 mvndep 0m 13s Maven dependency ordering for branch
        +1 mvninstall 8m 8s trunk passed
        +1 compile 2m 50s trunk passed
        +1 checkstyle 0m 58s trunk passed
        +1 mvnsite 4m 13s trunk passed
        +1 mvneclipse 0m 51s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 1m 9s trunk passed
        +1 javadoc 1m 46s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 2m 51s the patch passed
        +1 compile 2m 21s the patch passed
        -1 javac 2m 21s hadoop-yarn-project_hadoop-yarn generated 2 new + 33 unchanged - 2 fixed = 35 total (was 35)
        -1 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 144 new + 1437 unchanged - 158 fixed = 1581 total (was 1595)
        +1 mvnsite 3m 47s the patch passed
        +1 mvneclipse 0m 51s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        -1 findbugs 1m 32s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        -1 javadoc 1m 37s hadoop-yarn-project_hadoop-yarn generated 7 new + 6492 unchanged - 5 fixed = 6499 total (was 6497)
        -1 javadoc 0m 19s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 7 new + 933 unchanged - 5 fixed = 940 total (was 938)
        -1 unit 21m 9s hadoop-yarn in the patch failed.
        -1 unit 37m 36s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        94m 49s



        Reason Tests
        FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Nullcheck of node at line 1414 of value previously dereferenced in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(PlacementSet, boolean) At CapacityScheduler.java:1414 of value previously dereferenced in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(PlacementSet, boolean) At CapacityScheduler.java:[line 1414]
        Failed junit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
          hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices
          hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832229/YARN-5716.001.patch
        JIRA Issue YARN-5716
        Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
        uname Linux 7fb9449bde35 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6a38d11
        Default Java 1.8.0_101
        findbugs v3.0.0
        javac https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13319/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13319/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 12 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 8m 8s trunk passed +1 compile 2m 50s trunk passed +1 checkstyle 0m 58s trunk passed +1 mvnsite 4m 13s trunk passed +1 mvneclipse 0m 51s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 1m 9s trunk passed +1 javadoc 1m 46s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 2m 51s the patch passed +1 compile 2m 21s the patch passed -1 javac 2m 21s hadoop-yarn-project_hadoop-yarn generated 2 new + 33 unchanged - 2 fixed = 35 total (was 35) -1 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 144 new + 1437 unchanged - 158 fixed = 1581 total (was 1595) +1 mvnsite 3m 47s the patch passed +1 mvneclipse 0m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn -1 findbugs 1m 32s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 1m 37s hadoop-yarn-project_hadoop-yarn generated 7 new + 6492 unchanged - 5 fixed = 6499 total (was 6497) -1 javadoc 0m 19s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 7 new + 933 unchanged - 5 fixed = 940 total (was 938) -1 unit 21m 9s hadoop-yarn in the patch failed. -1 unit 37m 36s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 94m 49s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Nullcheck of node at line 1414 of value previously dereferenced in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(PlacementSet, boolean) At CapacityScheduler.java:1414 of value previously dereferenced in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(PlacementSet, boolean) At CapacityScheduler.java: [line 1414] Failed junit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832229/YARN-5716.001.patch JIRA Issue YARN-5716 Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux 7fb9449bde35 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6a38d11 Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt https://builds.apache.org/job/PreCommit-YARN-Build/13319/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13319/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13319/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached ver.2 patch.

        Show
        leftnoteasy Wangda Tan added a comment - Attached ver.2 patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 22s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 12 new or modified test files.
        0 mvndep 0m 13s Maven dependency ordering for branch
        +1 mvninstall 9m 16s trunk passed
        +1 compile 3m 18s trunk passed
        +1 checkstyle 1m 14s trunk passed
        +1 mvnsite 4m 5s trunk passed
        +1 mvneclipse 0m 46s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 0m 58s trunk passed
        +1 javadoc 1m 46s trunk passed
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 2m 49s the patch passed
        +1 compile 2m 23s the patch passed
        -1 javac 2m 23s hadoop-yarn-project_hadoop-yarn generated 1 new + 34 unchanged - 1 fixed = 35 total (was 35)
        -1 checkstyle 0m 57s hadoop-yarn-project/hadoop-yarn: The patch generated 145 new + 1427 unchanged - 164 fixed = 1572 total (was 1591)
        +1 mvnsite 4m 1s the patch passed
        +1 mvneclipse 0m 47s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 1m 10s the patch passed
        +1 javadoc 1m 27s hadoop-yarn-project_hadoop-yarn generated 0 new + 6487 unchanged - 10 fixed = 6487 total (was 6497)
        +1 javadoc 0m 20s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938)
        -1 unit 21m 33s hadoop-yarn in the patch failed.
        +1 unit 35m 51s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        94m 36s



        Reason Tests
        Failed junit tests hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832550/YARN-5716.002.patch
        JIRA Issue YARN-5716
        Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
        uname Linux c38a9aafbae2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 96b1266
        Default Java 1.8.0_101
        findbugs v3.0.0
        javac https://builds.apache.org/job/PreCommit-YARN-Build/13342/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13342/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13342/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13342/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13342/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13342/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 12 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 9m 16s trunk passed +1 compile 3m 18s trunk passed +1 checkstyle 1m 14s trunk passed +1 mvnsite 4m 5s trunk passed +1 mvneclipse 0m 46s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 58s trunk passed +1 javadoc 1m 46s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 2m 49s the patch passed +1 compile 2m 23s the patch passed -1 javac 2m 23s hadoop-yarn-project_hadoop-yarn generated 1 new + 34 unchanged - 1 fixed = 35 total (was 35) -1 checkstyle 0m 57s hadoop-yarn-project/hadoop-yarn: The patch generated 145 new + 1427 unchanged - 164 fixed = 1572 total (was 1591) +1 mvnsite 4m 1s the patch passed +1 mvneclipse 0m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 1m 10s the patch passed +1 javadoc 1m 27s hadoop-yarn-project_hadoop-yarn generated 0 new + 6487 unchanged - 10 fixed = 6487 total (was 6497) +1 javadoc 0m 20s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938) -1 unit 21m 33s hadoop-yarn in the patch failed. +1 unit 35m 51s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 94m 36s Reason Tests Failed junit tests hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832550/YARN-5716.002.patch JIRA Issue YARN-5716 Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux c38a9aafbae2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 96b1266 Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/13342/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13342/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13342/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13342/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13342/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13342/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jianhe Jian He added a comment -
        • why volatile
              private volatile BlockingQueue<ResourceCommitRequest<FiCaSchedulerApp, FiCaSchedulerNode>>
                  backlogs = new LinkedBlockingQueue<>();
          
        • 1 millisecond ? is this intentional ?
                      // Don't run schedule if we have some pending backlogs already
                      if (cs.getAsyncSchedulingPendingBacklogs() > 100) {
                        Thread.sleep(1);
          
        • getReservedSchedulerKey vs getAllocatedSchedulerKey: looks like the same ?
        • allocateContainersToNode: remove withNodeHeartbeat flag. Make the other caller call "allocateContainersToNode(PlacementSet<FiCaSchedulerNode> ps" directly
        • "allocateContainersToNode(PlacementSet<FiCaSchedulerNode> ps ": maybe use lookAtSingleNode boolean to fork the code path. Make a separate method.
        • the first if condition is probably not needed.
              if (null != allocated || null != reserved || (null != released && !released
                  .isEmpty())) {
                List<ContainerAllocationContext<FiCaSchedulerApp, FiCaSchedulerNode>>
                    allocationsList = null;
                if (allocated != null) {
                  allocationsList = new ArrayList<>();
                  allocationsList.add(allocated);
                }
          
                List<ContainerAllocationContext<FiCaSchedulerApp, FiCaSchedulerNode>>
                    reservationsList = null;
                if (reserved != null) {
                  reservationsList = new ArrayList<>();
                  reservationsList.add(reserved);
                }
          
                return new ResourceCommitRequest<>(allocationsList, reservationsList,
                    released);
              }
          
        Show
        jianhe Jian He added a comment - why volatile private volatile BlockingQueue<ResourceCommitRequest<FiCaSchedulerApp, FiCaSchedulerNode>> backlogs = new LinkedBlockingQueue<>(); 1 millisecond ? is this intentional ? // Don't run schedule if we have some pending backlogs already if (cs.getAsyncSchedulingPendingBacklogs() > 100) { Thread .sleep(1); getReservedSchedulerKey vs getAllocatedSchedulerKey: looks like the same ? allocateContainersToNode: remove withNodeHeartbeat flag. Make the other caller call "allocateContainersToNode(PlacementSet<FiCaSchedulerNode> ps" directly "allocateContainersToNode(PlacementSet<FiCaSchedulerNode> ps ": maybe use lookAtSingleNode boolean to fork the code path. Make a separate method. the first if condition is probably not needed. if ( null != allocated || null != reserved || ( null != released && !released .isEmpty())) { List<ContainerAllocationContext<FiCaSchedulerApp, FiCaSchedulerNode>> allocationsList = null ; if (allocated != null ) { allocationsList = new ArrayList<>(); allocationsList.add(allocated); } List<ContainerAllocationContext<FiCaSchedulerApp, FiCaSchedulerNode>> reservationsList = null ; if (reserved != null ) { reservationsList = new ArrayList<>(); reservationsList.add(reserved); } return new ResourceCommitRequest<>(allocationsList, reservationsList, released); }
        Hide
        jianhe Jian He added a comment -
        • Fix unnecessary format changes in CapacityScheduler
        • Below null check is unnecessary
                      RMContainer fromReservedContainer = null;
                      if (allocation.getAllocateFromReservedContainer() != null) {
                        fromReservedContainer =
                            allocation.getAllocateFromReservedContainer()
                                .getRmContainer();
                      }
          
        • This seems inconsistent with following code which will not add reserved resource back.
                    // Do we have enough space on this node?
                    Resource availableResource = Resources.clone(
                        schedulerContainer.getSchedulerNode().getUnallocatedResource());
                    if (allocation.getAllocateFromReservedContainer() != null) {
                      Resources.addTo(availableResource,
                          allocation.getAllocateFromReservedContainer()
                              .getRmContainer().getReservedResource());
                    }
          
        • may be we can check anythingAllocatedOrReserved and getAllocateFromReservedContainer outside of accept so that they do need to be checked for every queue in the path.
              boolean accepted = true;
              if (!reReservation) {
                // Check parent
                accepted = getCSLeafQueue().accept(cluster, request);
              }
          
          
        • should these methods be synchronized for consistency
            public void incUnconfirmedRes(Resource res) {
              unconfirmedAllocatedMem.addAndGet(res.getMemorySize());
              unconfirmedAllocatedVcores.addAndGet(res.getVirtualCores());
            }
          
            public void decUnconfirmedRes(Resource res) {
              unconfirmedAllocatedMem.addAndGet(-res.getMemorySize());
              unconfirmedAllocatedVcores.addAndGet(-res.getVirtualCores());
            }
           
        • Too many data structures which wrapps container related info with similar names: CSAssignment, ContainerAllocation, ContainerAllocationContext, SchedulerContainer.. could you consolidate these ?
        • SchedulingPlacementSet has a bunch unused methods and introduced unused class ResourceRequestUpdateResult, can we add them when needed ?
        • rename allocationResult.updatedContainer properly ? or add comments to what this field is used for
        • Add comments to what preCheckForPlacementSet does
        • preCheckForSingleNode -> checkIfNodeBlackListed
        • change getFinishedStatus to isCompleted
        • I think this needs to be done in accept ?
              // Inform the application it is about to get a scheduling opportunity
              // TODO, we may need to revisit here to see if we should add scheduling
              // opportunity here
              application.addSchedulingOpportunity(schedulerKey);
           
        • FiCaSchedulerApp#toBeRemovedIncRequests, why is this field moved to this class ? now the toBeRemovedIncRequests will be removed on allocate API only
        Show
        jianhe Jian He added a comment - Fix unnecessary format changes in CapacityScheduler Below null check is unnecessary RMContainer fromReservedContainer = null ; if (allocation.getAllocateFromReservedContainer() != null ) { fromReservedContainer = allocation.getAllocateFromReservedContainer() .getRmContainer(); } This seems inconsistent with following code which will not add reserved resource back. // Do we have enough space on this node? Resource availableResource = Resources.clone( schedulerContainer.getSchedulerNode().getUnallocatedResource()); if (allocation.getAllocateFromReservedContainer() != null ) { Resources.addTo(availableResource, allocation.getAllocateFromReservedContainer() .getRmContainer().getReservedResource()); } may be we can check anythingAllocatedOrReserved and getAllocateFromReservedContainer outside of accept so that they do need to be checked for every queue in the path. boolean accepted = true ; if (!reReservation) { // Check parent accepted = getCSLeafQueue().accept(cluster, request); } should these methods be synchronized for consistency public void incUnconfirmedRes(Resource res) { unconfirmedAllocatedMem.addAndGet(res.getMemorySize()); unconfirmedAllocatedVcores.addAndGet(res.getVirtualCores()); } public void decUnconfirmedRes(Resource res) { unconfirmedAllocatedMem.addAndGet(-res.getMemorySize()); unconfirmedAllocatedVcores.addAndGet(-res.getVirtualCores()); } Too many data structures which wrapps container related info with similar names: CSAssignment, ContainerAllocation, ContainerAllocationContext, SchedulerContainer.. could you consolidate these ? SchedulingPlacementSet has a bunch unused methods and introduced unused class ResourceRequestUpdateResult, can we add them when needed ? rename allocationResult.updatedContainer properly ? or add comments to what this field is used for Add comments to what preCheckForPlacementSet does preCheckForSingleNode -> checkIfNodeBlackListed change getFinishedStatus to isCompleted I think this needs to be done in accept ? // Inform the application it is about to get a scheduling opportunity // TODO, we may need to revisit here to see if we should add scheduling // opportunity here application.addSchedulingOpportunity(schedulerKey); FiCaSchedulerApp#toBeRemovedIncRequests, why is this field moved to this class ? now the toBeRemovedIncRequests will be removed on allocate API only
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Jian He for reviews,

        1 millisecond ? is this intentional ?

        This is to avoid busy loop and pinned on CPU, sleep 1 ms can make CPU usage close to 0%.

        getReservedSchedulerKey vs getAllocatedSchedulerKey: looks like the same ?

        Yeah they're similiar, since they're not new from this patch, I prefer to address this separately.

        Too many data structures which wrapps container related info with similar names: CSAssignment, ContainerAllocation, ContainerAllocationContext, SchedulerContainer.. could you consolidate these ?

        Agree, but I prefer to address them separately, CSAssignment/ContainerAllocation are not introduced by this patch. Relation between ContainerAllocationContext and SchedulerContainer is has-more, so we don't need to consolidate the latter two.

        SchedulingPlacementSet has a bunch unused methods and introduced unused class ResourceRequestUpdateResult, can we add them when needed ?

        That is part of the SchedulingPlacementSet definition, I prefer to keep them here. My next patch will start using it.

        Addressed all other comments. (ver.3)

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Jian He for reviews, 1 millisecond ? is this intentional ? This is to avoid busy loop and pinned on CPU, sleep 1 ms can make CPU usage close to 0%. getReservedSchedulerKey vs getAllocatedSchedulerKey: looks like the same ? Yeah they're similiar, since they're not new from this patch, I prefer to address this separately. Too many data structures which wrapps container related info with similar names: CSAssignment, ContainerAllocation, ContainerAllocationContext, SchedulerContainer.. could you consolidate these ? Agree, but I prefer to address them separately, CSAssignment/ContainerAllocation are not introduced by this patch. Relation between ContainerAllocationContext and SchedulerContainer is has-more, so we don't need to consolidate the latter two. SchedulingPlacementSet has a bunch unused methods and introduced unused class ResourceRequestUpdateResult, can we add them when needed ? That is part of the SchedulingPlacementSet definition, I prefer to keep them here. My next patch will start using it. Addressed all other comments. (ver.3)
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 6m 51s trunk passed
        +1 compile 2m 31s trunk passed
        +1 checkstyle 1m 0s trunk passed
        +1 mvnsite 3m 41s trunk passed
        +1 mvneclipse 0m 47s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 0m 55s trunk passed
        +1 javadoc 1m 55s trunk passed
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 2m 52s the patch passed
        +1 compile 2m 18s the patch passed
        -1 javac 2m 18s hadoop-yarn-project_hadoop-yarn generated 1 new + 34 unchanged - 1 fixed = 35 total (was 35)
        -1 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 141 new + 1471 unchanged - 165 fixed = 1612 total (was 1636)
        +1 mvnsite 3m 29s the patch passed
        +1 mvneclipse 0m 44s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 1m 2s the patch passed
        +1 javadoc 1m 20s hadoop-yarn-project_hadoop-yarn generated 0 new + 6484 unchanged - 10 fixed = 6484 total (was 6494)
        +1 javadoc 0m 18s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938)
        -1 unit 21m 32s hadoop-yarn in the patch failed.
        -1 unit 39m 19s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        93m 31s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth
          hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices
          hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834066/YARN-5716.003.patch
        JIRA Issue YARN-5716
        Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
        uname Linux 357ab1b6d970 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c62ae71
        Default Java 1.8.0_101
        findbugs v3.0.0
        javac https://builds.apache.org/job/PreCommit-YARN-Build/13435/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13435/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13435/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13435/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13435/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt https://builds.apache.org/job/PreCommit-YARN-Build/13435/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13435/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13435/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 51s trunk passed +1 compile 2m 31s trunk passed +1 checkstyle 1m 0s trunk passed +1 mvnsite 3m 41s trunk passed +1 mvneclipse 0m 47s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 55s trunk passed +1 javadoc 1m 55s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 2m 52s the patch passed +1 compile 2m 18s the patch passed -1 javac 2m 18s hadoop-yarn-project_hadoop-yarn generated 1 new + 34 unchanged - 1 fixed = 35 total (was 35) -1 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 141 new + 1471 unchanged - 165 fixed = 1612 total (was 1636) +1 mvnsite 3m 29s the patch passed +1 mvneclipse 0m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 1m 2s the patch passed +1 javadoc 1m 20s hadoop-yarn-project_hadoop-yarn generated 0 new + 6484 unchanged - 10 fixed = 6484 total (was 6494) +1 javadoc 0m 18s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938) -1 unit 21m 32s hadoop-yarn in the patch failed. -1 unit 39m 19s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 93m 31s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices   hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834066/YARN-5716.003.patch JIRA Issue YARN-5716 Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux 357ab1b6d970 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c62ae71 Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/13435/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13435/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13435/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13435/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13435/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt https://builds.apache.org/job/PreCommit-YARN-Build/13435/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13435/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13435/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached ver.4 patch, fixed UT failures and javac warnings.

        Show
        leftnoteasy Wangda Tan added a comment - Attached ver.4 patch, fixed UT failures and javac warnings.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
        0 mvndep 0m 56s Maven dependency ordering for branch
        +1 mvninstall 7m 45s trunk passed
        +1 compile 2m 17s trunk passed
        +1 checkstyle 0m 59s trunk passed
        +1 mvnsite 3m 33s trunk passed
        +1 mvneclipse 0m 47s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 0m 56s trunk passed
        +1 javadoc 1m 42s trunk passed
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 2m 45s the patch passed
        +1 compile 2m 14s the patch passed
        +1 javac 2m 14s the patch passed
        -1 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 140 new + 1476 unchanged - 164 fixed = 1616 total (was 1640)
        +1 mvnsite 3m 29s the patch passed
        +1 mvneclipse 0m 42s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 1m 3s the patch passed
        +1 javadoc 1m 20s hadoop-yarn-project_hadoop-yarn generated 0 new + 6484 unchanged - 10 fixed = 6484 total (was 6494)
        +1 javadoc 0m 18s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938)
        -1 unit 20m 32s hadoop-yarn in the patch failed.
        +1 unit 35m 56s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        89m 46s



        Reason Tests
        Failed junit tests hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834252/YARN-5716.004.patch
        JIRA Issue YARN-5716
        Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
        uname Linux c335eb7b42f7 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e9c4616
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13446/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13446/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13446/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13446/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13446/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files. 0 mvndep 0m 56s Maven dependency ordering for branch +1 mvninstall 7m 45s trunk passed +1 compile 2m 17s trunk passed +1 checkstyle 0m 59s trunk passed +1 mvnsite 3m 33s trunk passed +1 mvneclipse 0m 47s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 56s trunk passed +1 javadoc 1m 42s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 2m 45s the patch passed +1 compile 2m 14s the patch passed +1 javac 2m 14s the patch passed -1 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 140 new + 1476 unchanged - 164 fixed = 1616 total (was 1640) +1 mvnsite 3m 29s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 1m 3s the patch passed +1 javadoc 1m 20s hadoop-yarn-project_hadoop-yarn generated 0 new + 6484 unchanged - 10 fixed = 6484 total (was 6494) +1 javadoc 0m 18s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938) -1 unit 20m 32s hadoop-yarn in the patch failed. +1 unit 35m 56s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 89m 46s Reason Tests Failed junit tests hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834252/YARN-5716.004.patch JIRA Issue YARN-5716 Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux c335eb7b42f7 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e9c4616 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13446/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13446/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13446/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13446/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13446/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jianhe Jian He added a comment -
        • remove this?
                  /*
                  FIXME
                  application
                      .getCSLeafQueue()
                      .getOrderingPolicy()
                      .containerAllocated(application,
                          application.getRMContainer(updatedContainer.getId()));
                  */
          
        • When stop, if the thread is interrupted, the else condition will make the thread continue to run. the join in stop will always timeout.
                    if (!runSchedules.get() && !Thread.currentThread().isInterrupted()) {
                      Thread.sleep(100);
                    } else{
                      // Don't run schedule if we have some pending backlogs already
                      if (cs.getAsyncSchedulingPendingBacklogs() > 100) {
                        Thread.sleep(1);
                      } else{
                        schedule(cs);
                      }
                    }
                  } catch (InterruptedException ie) {
                    // Do nothing
                  }
          
        • the withNodeHeartbeat flag is not needed, because it's already checked in the caller
            private boolean canAllocateMore(CSAssignment assignment,
                boolean withNodeHeartbeat) {
          
        • how about check whether "getNode(node.getNodeID())" equals to null, I feel that's easier to reason for a removed node
              // Backward compatible way to make sure previous behavior which allocation
              // driven by node heartbeat works.
              if (getNode(node.getNodeID()) != node)
          
        • This if condition can be merged into previous "if (reservedContainer != null) {" condition, as they are the same.
              // Do not schedule if there are any reservations to fulfill on the node
              if (node.getReservedContainer() != null) {
                if (LOG.isDebugEnabled()) {
                  LOG.debug("Skipping scheduling since node " + node.getNodeID()
                      + " is reserved by application " + node.getReservedContainer()
                      .getContainerId().getApplicationAttemptId());
                }
                return null;
              }
           
        • canSchedule flag is not needed, the previuous if condition can just return direclty
               if (!canSchedule) {
                if (LOG.isDebugEnabled()) {
                  LOG.debug("This node or this node partition doesn't have available or"
                      + "killable resource");
                }
                return null;
              }
          
        • the canSchedule flag here is also not needed
          similarly,
              boolean canSchedule = false;
          
              // When this time look at multiple nodes, try schedule if the
              // partition has any available resource
              if (root.getQueueCapacities().getUsedCapacity(ps.getPartition()) < 1.0f
                  || preemptionManager.getKillableResource(
                  CapacitySchedulerConfiguration.ROOT, ps.getPartition()) != Resources
                  .none()) {
                canSchedule = true;
              }
          
              if (!canSchedule) {
                if (LOG.isDebugEnabled()) {
                  LOG.debug("This node or this node partition doesn't have available or"
                      + "killable resource");
                }
                return null;
              }
          
        • Looks like one behavior change is that previously on node heartbeat, we always satisfy reservedContainer first, now in async scheduling, it's not the case any more ?
        • this if check is not needed, beacuse it's already checked in the caller
            private CSAssignment allocateContainersOnMultiNodes(
                PlacementSet<FiCaSchedulerNode> ps) {
              if (rmContext.isWorkPreservingRecoveryEnabled() && !rmContext
                  .isSchedulerReadyForAllocatingContainers()) {
                return null;
              }
          
        • PlacementSet, placement is an abstract name, how about NodeSet to be more concret?
          -PlacementSetUtils.getSingleNode -> hasSingleNode
        • ContainerAllocationContext -> ContainerAllocationProposal
        • nodePartition parameter is not needed, it can be inferred from 'node' parameter
            public SchedulerContainer(A app, N node, RMContainer rmContainer,
                String nodePartition, boolean allocated) {
          
        • remove this method, a new method with the same name just makes it more confusing. And the other getSchedulerContainer method with false condition is already being used in some places.
            private SchedulerContainer<FiCaSchedulerApp, FiCaSchedulerNode> getSchedulerContainer(
                RMContainer rmContainer) {
              return getSchedulerContainer(rmContainer, false);
            }
          
        • These two if conditidions code are duplicated, can be moved out side
                    if (updateUnconfirmedAllocatedResource) {
                      app.decUnconfirmedRes(request.getTotalAllocatedResource());
                    }
                    LOG.info("Allocation proposal accepted");
                  } else{
                    if (updateUnconfirmedAllocatedResource) {
                      app.decUnconfirmedRes(request.getTotalAllocatedResource());
                    }
          
        • remove this
          // If we do assign, remove the queue and re-insert in-order to re-sort\
        • In LeafQueue#updateCurrentResourceLimits, multiple threads will update cachedResourceLimitsForHeadroom without synchronization
        • Fix indetation
                if (null != application) {
                ActivitiesLogger.APP.startAppAllocationRecording(activitiesManager,
                    node.getNodeID(), SystemClock.getInstance().getTime(), application);
                CSAssignment assignment = application.assignContainers(clusterResource,
                    ps, currentResourceLimits, schedulingMode, reservedContainer);
                return assignment;
                }
          
        • LeafQueeu#handleReservedContainer -> allocateReservedContainer
        • SchedulerApplicationAttempt#incNumAllocatedContainers, all the locality statistics functionality are removed ?
        Show
        jianhe Jian He added a comment - remove this? /* FIXME application .getCSLeafQueue() .getOrderingPolicy() .containerAllocated(application, application.getRMContainer(updatedContainer.getId())); */ When stop, if the thread is interrupted, the else condition will make the thread continue to run. the join in stop will always timeout. if (!runSchedules.get() && ! Thread .currentThread().isInterrupted()) { Thread .sleep(100); } else { // Don't run schedule if we have some pending backlogs already if (cs.getAsyncSchedulingPendingBacklogs() > 100) { Thread .sleep(1); } else { schedule(cs); } } } catch (InterruptedException ie) { // Do nothing } the withNodeHeartbeat flag is not needed, because it's already checked in the caller private boolean canAllocateMore(CSAssignment assignment, boolean withNodeHeartbeat) { how about check whether "getNode(node.getNodeID())" equals to null, I feel that's easier to reason for a removed node // Backward compatible way to make sure previous behavior which allocation // driven by node heartbeat works. if (getNode(node.getNodeID()) != node) This if condition can be merged into previous "if (reservedContainer != null) {" condition, as they are the same. // Do not schedule if there are any reservations to fulfill on the node if (node.getReservedContainer() != null ) { if (LOG.isDebugEnabled()) { LOG.debug( "Skipping scheduling since node " + node.getNodeID() + " is reserved by application " + node.getReservedContainer() .getContainerId().getApplicationAttemptId()); } return null ; } canSchedule flag is not needed, the previuous if condition can just return direclty if (!canSchedule) { if (LOG.isDebugEnabled()) { LOG.debug( "This node or this node partition doesn't have available or" + "killable resource" ); } return null ; } the canSchedule flag here is also not needed similarly, boolean canSchedule = false ; // When this time look at multiple nodes, try schedule if the // partition has any available resource if (root.getQueueCapacities().getUsedCapacity(ps.getPartition()) < 1.0f || preemptionManager.getKillableResource( CapacitySchedulerConfiguration.ROOT, ps.getPartition()) != Resources .none()) { canSchedule = true ; } if (!canSchedule) { if (LOG.isDebugEnabled()) { LOG.debug( "This node or this node partition doesn't have available or" + "killable resource" ); } return null ; } Looks like one behavior change is that previously on node heartbeat, we always satisfy reservedContainer first, now in async scheduling, it's not the case any more ? this if check is not needed, beacuse it's already checked in the caller private CSAssignment allocateContainersOnMultiNodes( PlacementSet<FiCaSchedulerNode> ps) { if (rmContext.isWorkPreservingRecoveryEnabled() && !rmContext .isSchedulerReadyForAllocatingContainers()) { return null ; } PlacementSet, placement is an abstract name, how about NodeSet to be more concret? -PlacementSetUtils.getSingleNode -> hasSingleNode ContainerAllocationContext -> ContainerAllocationProposal nodePartition parameter is not needed, it can be inferred from 'node' parameter public SchedulerContainer(A app, N node, RMContainer rmContainer, String nodePartition, boolean allocated) { remove this method, a new method with the same name just makes it more confusing. And the other getSchedulerContainer method with false condition is already being used in some places. private SchedulerContainer<FiCaSchedulerApp, FiCaSchedulerNode> getSchedulerContainer( RMContainer rmContainer) { return getSchedulerContainer(rmContainer, false ); } These two if conditidions code are duplicated, can be moved out side if (updateUnconfirmedAllocatedResource) { app.decUnconfirmedRes(request.getTotalAllocatedResource()); } LOG.info( "Allocation proposal accepted" ); } else { if (updateUnconfirmedAllocatedResource) { app.decUnconfirmedRes(request.getTotalAllocatedResource()); } remove this // If we do assign, remove the queue and re-insert in-order to re-sort \ In LeafQueue#updateCurrentResourceLimits, multiple threads will update cachedResourceLimitsForHeadroom without synchronization Fix indetation if ( null != application) { ActivitiesLogger.APP.startAppAllocationRecording(activitiesManager, node.getNodeID(), SystemClock.getInstance().getTime(), application); CSAssignment assignment = application.assignContainers(clusterResource, ps, currentResourceLimits, schedulingMode, reservedContainer); return assignment; } LeafQueeu#handleReservedContainer -> allocateReservedContainer SchedulerApplicationAttempt#incNumAllocatedContainers, all the locality statistics functionality are removed ?
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Jian He for the thorough review!

        how about check whether "getNode(node.getNodeID())" equals to null, I feel that's easier to reason for a removed node

        It is also possible that ref to the node is changed, for example, node resource updated. In that case, we may need to skip such node for safety.

        This if condition can be merged into previous "if (reservedContainer != null) {" condition, as they are the same.

        No we cannot do this merge, because it is possible that in the previous reservedContainer != null if, we reserve a new container so the check is not valid.

        Looks like one behavior change is that previously on node heartbeat, we always satisfy reservedContainer first, now in async scheduling, it's not the case any more ?

        It is still the same, if you look at allocateContainerOnSingleNode, we try to satisify reserved contaienr first.

        PlacementSet, placement is an abstract name, how about NodeSet to be more concret?

        I would prefer use "PlacementSet", since it is for "placement", we could add more information to it, for example, racks.

        PlacementSetUtils.getSingleNode -> hasSingleNode

        But what we need to do is to return the node if it is a single-node-placement-set, I think this name is better

        nodePartition parameter is not needed, it can be inferred from 'node' parameter

        The original purpose adding the partition is, the partition of node could be updated in between of proposal proposed and applied, it will be used to check if we should reject the proposal when partition of the node changed. I have a separate "TODO" in FiCaSchedulerApp:

        // TODO, make sure all node labels are not changed
        

        In LeafQueue#updateCurrentResourceLimits, multiple threads will update cachedResourceLimitsForHeadroom without synchronization

        This is intentional: we want the resourceLimitsForHeadroom is up-to-date, it is possible that one thread has some inconsistency data, but it will be corrected soon by other threads. Since the resourceLimitsForHeadroom is only used to just give more hints to application, it should be fine.
        And ResourceLimits is volatile, so it is safe as well.

        SchedulerApplicationAttempt#incNumAllocatedContainers, all the locality statistics functionality are removed ?

        Oh I missed that, will update it in the next iteration.

        Addressed all the other comments

        Uploaded patch ver.5

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Jian He for the thorough review! how about check whether "getNode(node.getNodeID())" equals to null, I feel that's easier to reason for a removed node It is also possible that ref to the node is changed, for example, node resource updated. In that case, we may need to skip such node for safety. This if condition can be merged into previous "if (reservedContainer != null) {" condition, as they are the same. No we cannot do this merge, because it is possible that in the previous reservedContainer != null if, we reserve a new container so the check is not valid. Looks like one behavior change is that previously on node heartbeat, we always satisfy reservedContainer first, now in async scheduling, it's not the case any more ? It is still the same, if you look at allocateContainerOnSingleNode , we try to satisify reserved contaienr first. PlacementSet, placement is an abstract name, how about NodeSet to be more concret? I would prefer use "PlacementSet", since it is for "placement", we could add more information to it, for example, racks. PlacementSetUtils.getSingleNode -> hasSingleNode But what we need to do is to return the node if it is a single-node-placement-set, I think this name is better nodePartition parameter is not needed, it can be inferred from 'node' parameter The original purpose adding the partition is, the partition of node could be updated in between of proposal proposed and applied, it will be used to check if we should reject the proposal when partition of the node changed. I have a separate "TODO" in FiCaSchedulerApp: // TODO, make sure all node labels are not changed In LeafQueue#updateCurrentResourceLimits, multiple threads will update cachedResourceLimitsForHeadroom without synchronization This is intentional: we want the resourceLimitsForHeadroom is up-to-date, it is possible that one thread has some inconsistency data, but it will be corrected soon by other threads. Since the resourceLimitsForHeadroom is only used to just give more hints to application, it should be fine. And ResourceLimits is volatile, so it is safe as well. SchedulerApplicationAttempt#incNumAllocatedContainers, all the locality statistics functionality are removed ? Oh I missed that, will update it in the next iteration. Addressed all the other comments Uploaded patch ver.5
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 23s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 6m 43s trunk passed
        +1 compile 2m 17s trunk passed
        +1 checkstyle 0m 59s trunk passed
        +1 mvnsite 3m 35s trunk passed
        +1 mvneclipse 0m 47s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 0m 57s trunk passed
        +1 javadoc 1m 45s trunk passed
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 3m 3s the patch passed
        +1 compile 2m 55s the patch passed
        +1 javac 2m 55s the patch passed
        -1 checkstyle 1m 0s hadoop-yarn-project/hadoop-yarn: The patch generated 132 new + 1472 unchanged - 164 fixed = 1604 total (was 1636)
        +1 mvnsite 4m 12s the patch passed
        +1 mvneclipse 0m 48s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 1m 19s the patch passed
        +1 javadoc 1m 41s hadoop-yarn-project_hadoop-yarn generated 0 new + 6484 unchanged - 10 fixed = 6484 total (was 6494)
        +1 javadoc 0m 22s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938)
        -1 unit 21m 51s hadoop-yarn in the patch failed.
        +1 unit 39m 41s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        95m 51s



        Reason Tests
        Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
          hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834568/YARN-5716.005.patch
        JIRA Issue YARN-5716
        Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
        uname Linux b555d595670c 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 262827c
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13459/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13459/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13459/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13459/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13459/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 43s trunk passed +1 compile 2m 17s trunk passed +1 checkstyle 0m 59s trunk passed +1 mvnsite 3m 35s trunk passed +1 mvneclipse 0m 47s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 57s trunk passed +1 javadoc 1m 45s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 3m 3s the patch passed +1 compile 2m 55s the patch passed +1 javac 2m 55s the patch passed -1 checkstyle 1m 0s hadoop-yarn-project/hadoop-yarn: The patch generated 132 new + 1472 unchanged - 164 fixed = 1604 total (was 1636) +1 mvnsite 4m 12s the patch passed +1 mvneclipse 0m 48s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 1m 19s the patch passed +1 javadoc 1m 41s hadoop-yarn-project_hadoop-yarn generated 0 new + 6484 unchanged - 10 fixed = 6484 total (was 6494) +1 javadoc 0m 22s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938) -1 unit 21m 51s hadoop-yarn in the patch failed. +1 unit 39m 41s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 95m 51s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834568/YARN-5716.005.patch JIRA Issue YARN-5716 Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux b555d595670c 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 262827c Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13459/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13459/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13459/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13459/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13459/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Rebased to latest trunk (006)

        Show
        leftnoteasy Wangda Tan added a comment - Rebased to latest trunk (006)
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 7m 51s trunk passed
        +1 compile 2m 20s trunk passed
        +1 checkstyle 0m 58s trunk passed
        +1 mvnsite 3m 40s trunk passed
        +1 mvneclipse 0m 50s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 0m 57s trunk passed
        +1 javadoc 1m 46s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 2m 49s the patch passed
        +1 compile 2m 24s the patch passed
        +1 javac 2m 24s the patch passed
        -1 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 131 new + 1467 unchanged - 164 fixed = 1598 total (was 1631)
        +1 mvnsite 3m 35s the patch passed
        +1 mvneclipse 0m 44s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 1m 6s the patch passed
        +1 javadoc 1m 27s hadoop-yarn-project_hadoop-yarn generated 0 new + 6484 unchanged - 10 fixed = 6484 total (was 6494)
        +1 javadoc 0m 20s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938)
        -1 unit 21m 37s hadoop-yarn in the patch failed.
        +1 unit 40m 17s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        95m 25s



        Reason Tests
        Failed junit tests hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834719/YARN-5716.006.patch
        JIRA Issue YARN-5716
        Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
        uname Linux d1b8fa5a5828 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f63cd78
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13467/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13467/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13467/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13467/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13467/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 7m 51s trunk passed +1 compile 2m 20s trunk passed +1 checkstyle 0m 58s trunk passed +1 mvnsite 3m 40s trunk passed +1 mvneclipse 0m 50s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 57s trunk passed +1 javadoc 1m 46s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 2m 49s the patch passed +1 compile 2m 24s the patch passed +1 javac 2m 24s the patch passed -1 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 131 new + 1467 unchanged - 164 fixed = 1598 total (was 1631) +1 mvnsite 3m 35s the patch passed +1 mvneclipse 0m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 1m 6s the patch passed +1 javadoc 1m 27s hadoop-yarn-project_hadoop-yarn generated 0 new + 6484 unchanged - 10 fixed = 6484 total (was 6494) +1 javadoc 0m 20s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938) -1 unit 21m 37s hadoop-yarn in the patch failed. +1 unit 40m 17s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 95m 25s Reason Tests Failed junit tests hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834719/YARN-5716.006.patch JIRA Issue YARN-5716 Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux d1b8fa5a5828 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f63cd78 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13467/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13467/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13467/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13467/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13467/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached ver.7 patch, fixed locality statistics and added one more test for that.

        Show
        leftnoteasy Wangda Tan added a comment - Attached ver.7 patch, fixed locality statistics and added one more test for that.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 7m 4s trunk passed
        +1 compile 2m 23s trunk passed
        +1 checkstyle 0m 57s trunk passed
        +1 mvnsite 3m 47s trunk passed
        +1 mvneclipse 0m 49s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 0m 58s trunk passed
        +1 javadoc 1m 46s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 2m 56s the patch passed
        +1 compile 2m 27s the patch passed
        +1 javac 2m 27s the patch passed
        -1 checkstyle 1m 0s hadoop-yarn-project/hadoop-yarn: The patch generated 140 new + 1467 unchanged - 164 fixed = 1607 total (was 1631)
        +1 mvnsite 3m 29s the patch passed
        +1 mvneclipse 0m 45s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 1m 4s the patch passed
        +1 javadoc 1m 22s hadoop-yarn-project_hadoop-yarn generated 0 new + 6484 unchanged - 10 fixed = 6484 total (was 6494)
        +1 javadoc 0m 18s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938)
        -1 unit 2m 31s hadoop-yarn in the patch failed.
        -1 unit 40m 56s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        76m 22s



        Reason Tests
        Failed junit tests hadoop.yarn.client.api.impl.TestTimelineClientV2Impl
          hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834748/YARN-5716.007.patch
        JIRA Issue YARN-5716
        Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
        uname Linux bc3ed0530d07 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 23d7d53
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13473/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13473/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13473/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13473/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt https://builds.apache.org/job/PreCommit-YARN-Build/13473/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13473/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13473/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 7m 4s trunk passed +1 compile 2m 23s trunk passed +1 checkstyle 0m 57s trunk passed +1 mvnsite 3m 47s trunk passed +1 mvneclipse 0m 49s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 58s trunk passed +1 javadoc 1m 46s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 2m 56s the patch passed +1 compile 2m 27s the patch passed +1 javac 2m 27s the patch passed -1 checkstyle 1m 0s hadoop-yarn-project/hadoop-yarn: The patch generated 140 new + 1467 unchanged - 164 fixed = 1607 total (was 1631) +1 mvnsite 3m 29s the patch passed +1 mvneclipse 0m 45s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 1m 4s the patch passed +1 javadoc 1m 22s hadoop-yarn-project_hadoop-yarn generated 0 new + 6484 unchanged - 10 fixed = 6484 total (was 6494) +1 javadoc 0m 18s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938) -1 unit 2m 31s hadoop-yarn in the patch failed. -1 unit 40m 56s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 76m 22s Reason Tests Failed junit tests hadoop.yarn.client.api.impl.TestTimelineClientV2Impl   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834748/YARN-5716.007.patch JIRA Issue YARN-5716 Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux bc3ed0530d07 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 23d7d53 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13473/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13473/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13473/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13473/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt https://builds.apache.org/job/PreCommit-YARN-Build/13473/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13473/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13473/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        latest patch looks good to me

        Show
        jianhe Jian He added a comment - latest patch looks good to me
        Hide
        sunilg Sunil G added a comment - - edited

        Sorry for pitching in late.

        Patch and approaches looks really great. Thank you.

        Some minor doubts/comments after a first round of scan

        1. is AppSchedulingInfo#updateMetricsForAllocatedContainer need to be under lock, I think its safely invoked within a lock block? Do you have some plans to reuse this module later somewhere, if so its fine.
        2. Every time AppSchedulingInfo#getSchedulingPlacementSet is invoked, a new SchedulingPlacementSet is created with all nodes iterator. Could we have an abstract class here so that we can maintain this code by keeping a basic impl for other non-used methods?
        3. LeafQueue#apply, could we have a more fine grained lock for allocateResource and orderingPolicy. I think we do not worry about request object, correct? pls correct me if I am wrong.
        4. Is it possible to keep the interfaces SchedulerContainer or ContainerAllocationProposal more read-only? I think it will help us keep interfaces more cleaner.
        5. One doubt in ResourceCommitRequest. Why are we adding allocated/reserved resources to totalReleasedResource? Sorry, i didnt understand here.
        I think i have only some more minor comments which i ll share in short while. Will make sure it ll not block the commit process

        Show
        sunilg Sunil G added a comment - - edited Sorry for pitching in late. Patch and approaches looks really great. Thank you. Some minor doubts/comments after a first round of scan 1. is AppSchedulingInfo#updateMetricsForAllocatedContainer need to be under lock, I think its safely invoked within a lock block? Do you have some plans to reuse this module later somewhere, if so its fine. 2. Every time AppSchedulingInfo#getSchedulingPlacementSet is invoked, a new SchedulingPlacementSet is created with all nodes iterator. Could we have an abstract class here so that we can maintain this code by keeping a basic impl for other non-used methods? 3. LeafQueue#apply, could we have a more fine grained lock for allocateResource and orderingPolicy. I think we do not worry about request object, correct? pls correct me if I am wrong. 4. Is it possible to keep the interfaces SchedulerContainer or ContainerAllocationProposal more read-only? I think it will help us keep interfaces more cleaner. 5. One doubt in ResourceCommitRequest. Why are we adding allocated/reserved resources to totalReleasedResource ? Sorry, i didnt understand here. I think i have only some more minor comments which i ll share in short while. Will make sure it ll not block the commit process
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Jian He!

        And thanks comments from Sunil G.

        For 1)/3).
        From my existing test result, the CPU time spent on write lock (which is under scheduler's accept/apply) is only a very small proportion comparing to the CPU time spent on read lock. It is around 3% to 10%. IAW, write lock is not a bottleneck, as of now, we cannot get a significant perf gain from the improvements in writelock. So I'd prefer to keep them as-is to just make it simpler.

        For 2). Yeah you're correct, it is a part of my next patch, I plan to move resource-request related functionalities from AppSchedulingInfo to SchedulingPlacementSet. I will share my POC patch soon.

        For 4). I'm not quite sure about what you meant, do you want to add comments to indicate it should be a read-only class or you want to remove writing APIs from these classes?

        For 5). I think you were talking about these logics:

            for (ContainerAllocationProposal<A,N> c : this.containersToReserve) {
              Resources.addTo(totalReservedResource,
                  c.getAllocatedOrReservedResource());
              for (SchedulerContainer<A,N> r : c.getToRelease()) {
                Resources.addTo(totalReleasedResource,
                    r.getRmContainer().getAllocatedOrReservedResource());
              }
            }
        

        For each allocated/reserved container, we can have a to-release containers : ContainerAllocationProposal#toRelease, which means we will kill/release these containers to allocate/reserve the given container. To distinguish these to-release containers from ResourceCommitRequest#toReleaseContainers. I call them "conditional release containers" : we will only release them when trying to allocate/reserve new resource.

        And I think we need to account both of conditional/unconditional to-release containers to the total release-able resource.

        Let me know if you have any comments.

        Thanks,

        For example continuous-reservation-looking/lazy-preemption.

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Jian He ! And thanks comments from Sunil G . For 1)/3). From my existing test result, the CPU time spent on write lock (which is under scheduler's accept/apply) is only a very small proportion comparing to the CPU time spent on read lock. It is around 3% to 10%. IAW, write lock is not a bottleneck, as of now, we cannot get a significant perf gain from the improvements in writelock. So I'd prefer to keep them as-is to just make it simpler. For 2). Yeah you're correct, it is a part of my next patch, I plan to move resource-request related functionalities from AppSchedulingInfo to SchedulingPlacementSet. I will share my POC patch soon. For 4). I'm not quite sure about what you meant, do you want to add comments to indicate it should be a read-only class or you want to remove writing APIs from these classes? For 5). I think you were talking about these logics: for (ContainerAllocationProposal<A,N> c : this .containersToReserve) { Resources.addTo(totalReservedResource, c.getAllocatedOrReservedResource()); for (SchedulerContainer<A,N> r : c.getToRelease()) { Resources.addTo(totalReleasedResource, r.getRmContainer().getAllocatedOrReservedResource()); } } For each allocated/reserved container, we can have a to-release containers : ContainerAllocationProposal#toRelease , which means we will kill/release these containers to allocate/reserve the given container. To distinguish these to-release containers from ResourceCommitRequest#toReleaseContainers . I call them "conditional release containers" : we will only release them when trying to allocate/reserve new resource. And I think we need to account both of conditional/unconditional to-release containers to the total release-able resource. Let me know if you have any comments. Thanks, For example continuous-reservation-looking/lazy-preemption.
        Hide
        sunilg Sunil G added a comment -

        Thanks Wangda Tan.

        do you want to add comments to indicate it should be a read-only class or you want to remove writing APIs from these classes?

        I was expecting to remove setter api's from this interface. Thoughts?

        continuous-reservation-looking

        I think the code is slightly complicated, but functionality seems fine. I am checking lazy-preemption now.

        Show
        sunilg Sunil G added a comment - Thanks Wangda Tan . do you want to add comments to indicate it should be a read-only class or you want to remove writing APIs from these classes? I was expecting to remove setter api's from this interface. Thoughts? continuous-reservation-looking I think the code is slightly complicated, but functionality seems fine. I am checking lazy-preemption now.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Sunil G, make sense.

        I just uploaded ver.8 patch, which removed all set interfaces from new added APIs. And I also added a comment to ResourceCommitRequest constructor to explain the to-release resource behavior.

        Show
        leftnoteasy Wangda Tan added a comment - Sunil G , make sense. I just uploaded ver.8 patch, which removed all set interfaces from new added APIs. And I also added a comment to ResourceCommitRequest constructor to explain the to-release resource behavior.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
        0 mvndep 0m 10s Maven dependency ordering for branch
        +1 mvninstall 7m 33s trunk passed
        +1 compile 2m 40s trunk passed
        +1 checkstyle 1m 3s trunk passed
        +1 mvnsite 4m 5s trunk passed
        +1 mvneclipse 0m 51s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 1m 7s trunk passed
        +1 javadoc 1m 59s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 3m 7s the patch passed
        +1 compile 2m 26s the patch passed
        +1 javac 2m 26s the patch passed
        -1 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 139 new + 1470 unchanged - 164 fixed = 1609 total (was 1634)
        +1 mvnsite 4m 5s the patch passed
        +1 mvneclipse 0m 45s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 1m 30s the patch passed
        +1 javadoc 1m 33s hadoop-yarn-project_hadoop-yarn generated 0 new + 6484 unchanged - 10 fixed = 6484 total (was 6494)
        +1 javadoc 0m 21s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938)
        -1 unit 66m 30s hadoop-yarn in the patch failed.
        +1 unit 35m 30s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        137m 56s



        Reason Tests
        Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
          hadoop.yarn.server.TestContainerManagerSecurity



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834987/YARN-5716.008.patch
        JIRA Issue YARN-5716
        Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
        uname Linux b28e8aadc4a2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b18f35f
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13491/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13491/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13491/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13491/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13491/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 7m 33s trunk passed +1 compile 2m 40s trunk passed +1 checkstyle 1m 3s trunk passed +1 mvnsite 4m 5s trunk passed +1 mvneclipse 0m 51s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 1m 7s trunk passed +1 javadoc 1m 59s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 3m 7s the patch passed +1 compile 2m 26s the patch passed +1 javac 2m 26s the patch passed -1 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 139 new + 1470 unchanged - 164 fixed = 1609 total (was 1634) +1 mvnsite 4m 5s the patch passed +1 mvneclipse 0m 45s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 1m 30s the patch passed +1 javadoc 1m 33s hadoop-yarn-project_hadoop-yarn generated 0 new + 6484 unchanged - 10 fixed = 6484 total (was 6494) +1 javadoc 0m 21s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938) -1 unit 66m 30s hadoop-yarn in the patch failed. +1 unit 35m 30s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 137m 56s Reason Tests Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestContainerManagerSecurity Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834987/YARN-5716.008.patch JIRA Issue YARN-5716 Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux b28e8aadc4a2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b18f35f Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13491/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13491/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13491/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13491/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13491/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Thanks Wangda Tan for the update.

        Few more general comments:

        1. computeUserLimitAndSetHeadroom is invoked invoked from accept call chain and from assigneContainers. I think its fine provided the both gives similar computed data. With user-limit preemption improvement, i think if compute is done outside these scheduler call flow, such user-limit computation costs can be brought down.
        2. In LeafQueue.accept,

                Resources.subtractFrom(usedResource,
                    request.getTotalReleasedResource());
        

        totalReleasedResource also include reserved resources too, correct. Do we need to decrement that? I am not very clear in this point here.

        3. In CS#allocateContainerOnSingleNode, node.getReservedContainer() is checked two times (second time to handle error case). Could we handle in first block itself.
        4. In FiCaSchedulerApp.accept,

        • a reference for resourceRequests is kept and under failure updated back to schedulingInfo object. I do not feel its a clean implementation. I remember this change, but I feel we need not have to take a reference in app level, may be we can keep it in schedulingInfo and can be used the same during recovery.
        • Method is very lengthy. We could take out logic like increase request etc. So it ll be more easier.
        • readLock is mostly needed for increased request and for appSchedulingInfo. I think some optimizations here could be done separately in another ticket as improvement.
        • preferring equals instead of if (fromReservedContainer != reservedContainerOnNode)

        Also I have checked reservation continue scheduling and lazy-preemption in particular in this round. Generally both looks fine for me.

        Show
        sunilg Sunil G added a comment - Thanks Wangda Tan for the update. Few more general comments: 1. computeUserLimitAndSetHeadroom is invoked invoked from accept call chain and from assigneContainers . I think its fine provided the both gives similar computed data. With user-limit preemption improvement, i think if compute is done outside these scheduler call flow, such user-limit computation costs can be brought down. 2. In LeafQueue.accept , Resources.subtractFrom(usedResource, request.getTotalReleasedResource()); totalReleasedResource also include reserved resources too, correct. Do we need to decrement that? I am not very clear in this point here. 3. In CS#allocateContainerOnSingleNode , node.getReservedContainer() is checked two times (second time to handle error case). Could we handle in first block itself. 4. In FiCaSchedulerApp.accept , a reference for resourceRequests is kept and under failure updated back to schedulingInfo object. I do not feel its a clean implementation. I remember this change, but I feel we need not have to take a reference in app level, may be we can keep it in schedulingInfo and can be used the same during recovery. Method is very lengthy. We could take out logic like increase request etc. So it ll be more easier. readLock is mostly needed for increased request and for appSchedulingInfo. I think some optimizations here could be done separately in another ticket as improvement. preferring equals instead of if (fromReservedContainer != reservedContainerOnNode) Also I have checked reservation continue scheduling and lazy-preemption in particular in this round. Generally both looks fine for me.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Sunil G for comments:

        For 1), yes it looks like we're doing UL check twice, however, the second time only check one application instead of thousands of apps.
        So the time spent on UL check for accept/apply is ignoreable from my performance tests.

        For 2), we will not check allocation-from-reserved container by the outter if (...) check

        For 3), Jian asked the same question, see my answer https://issues.apache.org/jira/browse/YARN-5716?focusedCommentId=15593292&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15593292:

        No we cannot do this merge, because it is possible that in the previous reservedContainer != null if, we reserve a new container so the check is not valid.

        For 4):

        a reference for resourceRequests is kept and under failure updated back to schedulingInfo object. I do not feel its a clean implementation...

        Actually we clone the ResourceRequest, and we need to check and make sure ResourceRequest is not changed or still required by the apps.

        Method is very lengthy. We could take out logic like increase request etc. So it ll be more easier.

        Done

        readLock is mostly needed for increased request and for appSchedulingInfo. I think some optimizations here could be done separately in another ticket as improvement.

        Yeah I agree, I would prefer to keep it as-is unless we find any performance issues.

        preferring equals instead of if (fromReservedContainer != reservedContainerOnNode)

        I intended to do that, I want to make sure two reserved container are pointed to the same instance.

        Please check ver.9 patch

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Sunil G for comments: For 1), yes it looks like we're doing UL check twice, however, the second time only check one application instead of thousands of apps. So the time spent on UL check for accept/apply is ignoreable from my performance tests. For 2), we will not check allocation-from-reserved container by the outter if (...) check For 3), Jian asked the same question, see my answer https://issues.apache.org/jira/browse/YARN-5716?focusedCommentId=15593292&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15593292: No we cannot do this merge, because it is possible that in the previous reservedContainer != null if, we reserve a new container so the check is not valid. For 4): a reference for resourceRequests is kept and under failure updated back to schedulingInfo object. I do not feel its a clean implementation... Actually we clone the ResourceRequest, and we need to check and make sure ResourceRequest is not changed or still required by the apps. Method is very lengthy. We could take out logic like increase request etc. So it ll be more easier. Done readLock is mostly needed for increased request and for appSchedulingInfo. I think some optimizations here could be done separately in another ticket as improvement. Yeah I agree, I would prefer to keep it as-is unless we find any performance issues. preferring equals instead of if (fromReservedContainer != reservedContainerOnNode) I intended to do that, I want to make sure two reserved container are pointed to the same instance. Please check ver.9 patch
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 6m 43s trunk passed
        +1 compile 2m 18s trunk passed
        +1 checkstyle 1m 0s trunk passed
        +1 mvnsite 3m 34s trunk passed
        +1 mvneclipse 0m 47s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 0m 58s trunk passed
        +1 javadoc 1m 43s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 2m 49s the patch passed
        +1 compile 2m 16s the patch passed
        +1 javac 2m 16s the patch passed
        -1 checkstyle 0m 59s hadoop-yarn-project/hadoop-yarn: The patch generated 142 new + 1467 unchanged - 164 fixed = 1609 total (was 1631)
        +1 mvnsite 3m 31s the patch passed
        +1 mvneclipse 0m 44s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 1m 4s the patch passed
        +1 javadoc 1m 22s hadoop-yarn-project_hadoop-yarn generated 0 new + 6484 unchanged - 10 fixed = 6484 total (was 6494)
        +1 javadoc 0m 19s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938)
        -1 unit 71m 54s hadoop-yarn in the patch failed.
        +1 unit 39m 54s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        143m 46s



        Reason Tests
        Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity
          hadoop.yarn.server.TestMiniYarnClusterNodeUtilization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835177/YARN-5716.009.patch
        JIRA Issue YARN-5716
        Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
        uname Linux f9982ae8c016 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c88c1dc
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13506/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13506/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13506/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13506/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13506/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 43s trunk passed +1 compile 2m 18s trunk passed +1 checkstyle 1m 0s trunk passed +1 mvnsite 3m 34s trunk passed +1 mvneclipse 0m 47s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 58s trunk passed +1 javadoc 1m 43s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 2m 49s the patch passed +1 compile 2m 16s the patch passed +1 javac 2m 16s the patch passed -1 checkstyle 0m 59s hadoop-yarn-project/hadoop-yarn: The patch generated 142 new + 1467 unchanged - 164 fixed = 1609 total (was 1631) +1 mvnsite 3m 31s the patch passed +1 mvneclipse 0m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 1m 4s the patch passed +1 javadoc 1m 22s hadoop-yarn-project_hadoop-yarn generated 0 new + 6484 unchanged - 10 fixed = 6484 total (was 6494) +1 javadoc 0m 19s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938) -1 unit 71m 54s hadoop-yarn in the patch failed. +1 unit 39m 54s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 143m 46s Reason Tests Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835177/YARN-5716.009.patch JIRA Issue YARN-5716 Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux f9982ae8c016 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c88c1dc Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13506/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13506/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13506/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13506/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13506/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Thanks Wangda Tan for the clarifications. Makes sense. New patch looks fine for me too.

        Show
        sunilg Sunil G added a comment - Thanks Wangda Tan for the clarifications. Makes sense. New patch looks fine for me too.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached ver.10 patch rebased to latest trunk, Rewrote tests for YARN-4963.

        Show
        leftnoteasy Wangda Tan added a comment - Attached ver.10 patch rebased to latest trunk, Rewrote tests for YARN-4963 .
        Hide
        leftnoteasy Wangda Tan added a comment -

        Ver.10 patch included unnecessary changes, attached ver.11.

        Show
        leftnoteasy Wangda Tan added a comment - Ver.10 patch included unnecessary changes, attached ver.11.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 14 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 6m 40s trunk passed
        +1 compile 2m 16s trunk passed
        +1 checkstyle 1m 0s trunk passed
        +1 mvnsite 3m 35s trunk passed
        +1 mvneclipse 0m 48s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 0m 59s trunk passed
        +1 javadoc 1m 50s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 2m 57s the patch passed
        +1 compile 2m 24s the patch passed
        +1 javac 2m 24s the patch passed
        -0 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 158 new + 1502 unchanged - 171 fixed = 1660 total (was 1673)
        +1 mvnsite 3m 41s the patch passed
        +1 mvneclipse 0m 48s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        -1 findbugs 1m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        +1 javadoc 1m 27s hadoop-yarn-project_hadoop-yarn generated 0 new + 6482 unchanged - 10 fixed = 6482 total (was 6492)
        +1 javadoc 0m 21s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938)
        -1 unit 65m 41s hadoop-yarn in the patch failed.
        -1 unit 36m 42s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        141m 34s



        Reason Tests
        FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateResourceRequests(List, boolean) makes inefficient use of keySet iterator instead of entrySet iterator At AppSchedulingInfo.java:of keySet iterator instead of entrySet iterator At AppSchedulingInfo.java:[line 428]
        Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing
          hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
          hadoop.yarn.server.TestContainerManagerSecurity
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-5716
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835890/YARN-5716.010.patch
        Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
        uname Linux f0063b63d53d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8a9388e
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13650/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13650/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13650/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13650/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13650/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13650/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 14 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 40s trunk passed +1 compile 2m 16s trunk passed +1 checkstyle 1m 0s trunk passed +1 mvnsite 3m 35s trunk passed +1 mvneclipse 0m 48s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 59s trunk passed +1 javadoc 1m 50s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 2m 57s the patch passed +1 compile 2m 24s the patch passed +1 javac 2m 24s the patch passed -0 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 158 new + 1502 unchanged - 171 fixed = 1660 total (was 1673) +1 mvnsite 3m 41s the patch passed +1 mvneclipse 0m 48s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn -1 findbugs 1m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 27s hadoop-yarn-project_hadoop-yarn generated 0 new + 6482 unchanged - 10 fixed = 6482 total (was 6492) +1 javadoc 0m 21s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938) -1 unit 65m 41s hadoop-yarn in the patch failed. -1 unit 36m 42s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 141m 34s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateResourceRequests(List, boolean) makes inefficient use of keySet iterator instead of entrySet iterator At AppSchedulingInfo.java:of keySet iterator instead of entrySet iterator At AppSchedulingInfo.java: [line 428] Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5716 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835890/YARN-5716.010.patch Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux f0063b63d53d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8a9388e Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13650/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13650/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/13650/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13650/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13650/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13650/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
        0 mvndep 0m 26s Maven dependency ordering for branch
        +1 mvninstall 6m 45s trunk passed
        +1 compile 2m 19s trunk passed
        +1 checkstyle 0m 58s trunk passed
        +1 mvnsite 3m 38s trunk passed
        +1 mvneclipse 0m 50s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 0m 58s trunk passed
        +1 javadoc 1m 45s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 2m 51s the patch passed
        +1 compile 2m 18s the patch passed
        +1 javac 2m 18s the patch passed
        -0 checkstyle 0m 59s hadoop-yarn-project/hadoop-yarn: The patch generated 147 new + 1464 unchanged - 169 fixed = 1611 total (was 1633)
        +1 mvnsite 3m 33s the patch passed
        +1 mvneclipse 0m 48s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 1m 4s the patch passed
        +1 javadoc 1m 24s hadoop-yarn-project_hadoop-yarn generated 0 new + 6482 unchanged - 10 fixed = 6482 total (was 6492)
        +1 javadoc 0m 20s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938)
        -1 unit 61m 7s hadoop-yarn in the patch failed.
        +1 unit 35m 35s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        135m 56s



        Reason Tests
        Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-5716
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835895/YARN-5716.011.patch
        Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
        uname Linux 25aaf9d628c4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / aa3cab1
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13674/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13674/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13674/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13674/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files. 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 6m 45s trunk passed +1 compile 2m 19s trunk passed +1 checkstyle 0m 58s trunk passed +1 mvnsite 3m 38s trunk passed +1 mvneclipse 0m 50s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 58s trunk passed +1 javadoc 1m 45s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 2m 51s the patch passed +1 compile 2m 18s the patch passed +1 javac 2m 18s the patch passed -0 checkstyle 0m 59s hadoop-yarn-project/hadoop-yarn: The patch generated 147 new + 1464 unchanged - 169 fixed = 1611 total (was 1633) +1 mvnsite 3m 33s the patch passed +1 mvneclipse 0m 48s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 1m 4s the patch passed +1 javadoc 1m 24s hadoop-yarn-project_hadoop-yarn generated 0 new + 6482 unchanged - 10 fixed = 6482 total (was 6492) +1 javadoc 0m 20s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 928 unchanged - 10 fixed = 928 total (was 938) -1 unit 61m 7s hadoop-yarn in the patch failed. +1 unit 35m 35s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 135m 56s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5716 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835895/YARN-5716.011.patch Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux 25aaf9d628c4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aa3cab1 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13674/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13674/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13674/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13674/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Rebased to latest trunk (v12).

        Show
        leftnoteasy Wangda Tan added a comment - Rebased to latest trunk (v12).
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 6m 54s trunk passed
        +1 compile 2m 25s trunk passed
        +1 checkstyle 1m 2s trunk passed
        +1 mvnsite 3m 46s trunk passed
        +1 mvneclipse 0m 50s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 0m 57s trunk passed
        +1 javadoc 1m 46s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        -1 mvninstall 1m 39s hadoop-yarn in the patch failed.
        -1 mvninstall 0m 19s hadoop-yarn-server-resourcemanager in the patch failed.
        -1 compile 1m 28s hadoop-yarn in the patch failed.
        -1 javac 1m 28s hadoop-yarn in the patch failed.
        -0 checkstyle 0m 57s hadoop-yarn-project/hadoop-yarn: The patch generated 146 new + 1466 unchanged - 169 fixed = 1612 total (was 1635)
        -1 mvnsite 1m 39s hadoop-yarn in the patch failed.
        -1 mvnsite 0m 21s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 mvneclipse 0m 47s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        -1 findbugs 0m 19s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 javadoc 1m 23s hadoop-yarn-project_hadoop-yarn generated 0 new + 6481 unchanged - 10 fixed = 6481 total (was 6491)
        +1 javadoc 0m 26s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 927 unchanged - 10 fixed = 927 total (was 937)
        -1 unit 26m 24s hadoop-yarn in the patch failed.
        -1 unit 0m 21s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        61m 56s



        Reason Tests
        Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManagerRegression
          hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-5716
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836242/YARN-5716.0012.patch
        Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
        uname Linux 9b167c1a587c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 90dd3a8
        Default Java 1.8.0_101
        findbugs v3.0.0
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn.txt
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        compile https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13709/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13709/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 54s trunk passed +1 compile 2m 25s trunk passed +1 checkstyle 1m 2s trunk passed +1 mvnsite 3m 46s trunk passed +1 mvneclipse 0m 50s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 57s trunk passed +1 javadoc 1m 46s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch -1 mvninstall 1m 39s hadoop-yarn in the patch failed. -1 mvninstall 0m 19s hadoop-yarn-server-resourcemanager in the patch failed. -1 compile 1m 28s hadoop-yarn in the patch failed. -1 javac 1m 28s hadoop-yarn in the patch failed. -0 checkstyle 0m 57s hadoop-yarn-project/hadoop-yarn: The patch generated 146 new + 1466 unchanged - 169 fixed = 1612 total (was 1635) -1 mvnsite 1m 39s hadoop-yarn in the patch failed. -1 mvnsite 0m 21s hadoop-yarn-server-resourcemanager in the patch failed. +1 mvneclipse 0m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn -1 findbugs 0m 19s hadoop-yarn-server-resourcemanager in the patch failed. +1 javadoc 1m 23s hadoop-yarn-project_hadoop-yarn generated 0 new + 6481 unchanged - 10 fixed = 6481 total (was 6491) +1 javadoc 0m 26s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 927 unchanged - 10 fixed = 927 total (was 937) -1 unit 26m 24s hadoop-yarn in the patch failed. -1 unit 0m 21s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 61m 56s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManagerRegression   hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5716 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836242/YARN-5716.0012.patch Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux 9b167c1a587c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 90dd3a8 Default Java 1.8.0_101 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13709/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13709/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13709/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Rebased again, (ver.13)

        Show
        leftnoteasy Wangda Tan added a comment - Rebased again, (ver.13)
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
        0 mvndep 0m 10s Maven dependency ordering for branch
        +1 mvninstall 7m 2s trunk passed
        +1 compile 2m 25s trunk passed
        +1 checkstyle 0m 57s trunk passed
        +1 mvnsite 3m 44s trunk passed
        +1 mvneclipse 0m 50s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 1m 4s trunk passed
        +1 javadoc 1m 51s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 3m 3s the patch passed
        +1 compile 2m 21s the patch passed
        +1 javac 2m 21s the patch passed
        -0 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 147 new + 1459 unchanged - 169 fixed = 1606 total (was 1628)
        +1 mvnsite 3m 46s the patch passed
        +1 mvneclipse 0m 49s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
        +1 findbugs 1m 10s the patch passed
        +1 javadoc 1m 28s hadoop-yarn-project_hadoop-yarn generated 0 new + 6481 unchanged - 10 fixed = 6481 total (was 6491)
        +1 javadoc 0m 21s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 927 unchanged - 10 fixed = 927 total (was 937)
        -1 unit 65m 48s hadoop-yarn in the patch failed.
        +1 unit 35m 56s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        141m 46s



        Reason Tests
        Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
          hadoop.yarn.server.TestContainerManagerSecurity



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-5716
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836395/YARN-5716.013.patch
        Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle
        uname Linux dded564b3a49 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 34173a4
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13737/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13737/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13737/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13737/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13737/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 7m 2s trunk passed +1 compile 2m 25s trunk passed +1 checkstyle 0m 57s trunk passed +1 mvnsite 3m 44s trunk passed +1 mvneclipse 0m 50s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 1m 4s trunk passed +1 javadoc 1m 51s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 3m 3s the patch passed +1 compile 2m 21s the patch passed +1 javac 2m 21s the patch passed -0 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 147 new + 1459 unchanged - 169 fixed = 1606 total (was 1628) +1 mvnsite 3m 46s the patch passed +1 mvneclipse 0m 49s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 1m 10s the patch passed +1 javadoc 1m 28s hadoop-yarn-project_hadoop-yarn generated 0 new + 6481 unchanged - 10 fixed = 6481 total (was 6491) +1 javadoc 0m 21s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 927 unchanged - 10 fixed = 927 total (was 937) -1 unit 65m 48s hadoop-yarn in the patch failed. +1 unit 35m 56s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 141m 46s Reason Tests Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestContainerManagerSecurity Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5716 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836395/YARN-5716.013.patch Optional Tests asflicense findbugs xml compile javac javadoc mvninstall mvnsite unit checkstyle uname Linux dded564b3a49 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 34173a4 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13737/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13737/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13737/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13737/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13737/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        we'll commit this soon, if no further comments

        Show
        jianhe Jian He added a comment - we'll commit this soon, if no further comments
        Hide
        jianhe Jian He added a comment -

        Committed to trunk , branch-2, thanks Wangda !

        Show
        jianhe Jian He added a comment - Committed to trunk , branch-2, thanks Wangda !
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10784 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10784/)
        YARN-5716. Add global scheduler interface definition and update (jianhe: rev de3b4aac561258ad242a3c5ed1c919428893fd4c)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/AssignmentInformation.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/IncreaseContainerAllocator.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/ContainerAllocationProposal.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/SchedulerContainer.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/activities/ActivitiesLogger.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/ResourceCommitRequest.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAsyncScheduling.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/ResourceRequestUpdateResult.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FairOrderingPolicy.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/PlacementSet.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/activities/ActivitiesManager.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/PlacementSetUtils.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FifoOrderingPolicyForPendingApps.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/ResourceAllocationCommitter.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FifoOrderingPolicy.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoAppAttempt.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/AbstractComparatorOrderingPolicy.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AbstractContainerAllocator.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesSchedulerActivities.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
        • (edit) hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/SchedulingPlacementSet.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/SimplePlacementSet.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerHealth.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10784 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10784/ ) YARN-5716 . Add global scheduler interface definition and update (jianhe: rev de3b4aac561258ad242a3c5ed1c919428893fd4c) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/AssignmentInformation.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/IncreaseContainerAllocator.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/ContainerAllocationProposal.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/SchedulerContainer.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/activities/ActivitiesLogger.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/ResourceCommitRequest.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAsyncScheduling.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/ResourceRequestUpdateResult.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FairOrderingPolicy.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/PlacementSet.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/activities/ActivitiesManager.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/PlacementSetUtils.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FifoOrderingPolicyForPendingApps.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/ResourceAllocationCommitter.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FifoOrderingPolicy.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoAppAttempt.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/AbstractComparatorOrderingPolicy.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AbstractContainerAllocator.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesSchedulerActivities.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java (edit) hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/SchedulingPlacementSet.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/SimplePlacementSet.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerHealth.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java

          People

          • Assignee:
            leftnoteasy Wangda Tan
            Reporter:
            leftnoteasy Wangda Tan
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development