Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3933

FairScheduler: Multiple calls to completedContainer are not safe

    Details

    • Hadoop Flags:
      Reviewed

      Description

      In our cluster we are seeing available memory and cores being negative.
      Initial inspection:
      Scenario no. 1:
      In capacity scheduler the method allocateContainersToNode() checks if
      there are excess reservation of containers for an application, and they are no longer needed then it calls queue.completedContainer() which causes resources being negative. And they were never assigned in the first place.

      I am still looking through the code. Can somebody suggest how to simulate excess containers assignments ?

      1. YARN-3933.001.patch
        12 kB
        Shiwei Guo
      2. YARN-3933.002.patch
        12 kB
        Shiwei Guo
      3. YARN-3933.003.patch
        12 kB
        Shiwei Guo
      4. YARN-3933.004.patch
        14 kB
        Sunil G
      5. YARN-3933.005.patch
        14 kB
        Sunil G
      6. YARN-3933.006.patch
        6 kB
        Miklos Szegedi
      7. yarn-3933-branch-2.8.patch
        6 kB
        Karthik Kambatla

        Issue Links

          Activity

          Hide
          guoshiwei Shiwei Guo added a comment -

          We also seeing this problems, and it may make the RM never allocate resource for a queue that has used negative resource.

          I did some research and found the this is mainly caused by a race condition of calling AbstractYarnScheduler.completedContainer. Lets take FairScheduler as an example:

          FairSchedular.java
          protected synchronized void completedContainer(RMContainer rmContainer,
                ContainerStatus containerStatus, RMContainerEventType event) {
              if (rmContainer == null) {
                LOG.info("Null container completed...");
                return;
              }
          
          
              Container container = rmContainer.getContainer();
          
              // Get the application for the finished container
              FSAppAttempt application =
                  getCurrentAttemptForContainer(container.getId());
              ApplicationId appId =
                  container.getId().getApplicationAttemptId().getApplicationId();
              if (application == null) {
                LOG.info("Container " + container + " of" +
                    " unknown application attempt " + appId +
                    " completed with event " + event);
                return;
              }
              if(!application.getLiveContainersMap().containsKey(container.getId())){
                LOG.info("Container " + container + " of application attempt " + appId
                + " is not alive, skip do completedContainer operation on event " + event);
                return;
              }
          
              // Get the node on which the container was allocated
              FSSchedulerNode node = getFSSchedulerNode(container.getNodeId());
          
              if (rmContainer.getState() == RMContainerState.RESERVED) {
                application.unreserve(rmContainer.getReservedPriority(), node);
              } else {
                application.containerCompleted(rmContainer, containerStatus, event);
                node.releaseContainer(container);
                updateRootQueueMetrics();
              }
          
              LOG.info("Application attempt " + application.getApplicationAttemptId()
                  + " released container " + container.getId() + " on node: " + node
                  + " with event: " + event);
            }
          

          completedContainer method will call application.containerCompleted, which will subtraction the resources used by this container from the usedResource counter of the application. So, if the completedContainer are called twice with the same container, the counter is subtracted too much values. So is the updateRootQueueMetrics call, so we can see negative allocatedMemory on rootQueue.

          The solution is to check whether the container being supplied is still live inside the completedContainer (as shown in the patch). There is some check before calling completedContainer, but that's not enough.

          For a more deeply discussion, the completedContainer may be called from two place:

          1. Trigered by RMContainerEventType.FINISHED event:

          FairScheduler.nodeUpdate
          // Process completed containers
              for (ContainerStatus completedContainer : completedContainers) {
                ContainerId containerId = completedContainer.getContainerId();
                LOG.debug("Container FINISHED: " + containerId);
                completedContainer(getRMContainer(containerId),
                    completedContainer, RMContainerEventType.FINISHED);
              }
          

          2. Trigered by RMContainerEventType.RELEASED

          AbstractYarnScheduler.releaseContainers
          completedContainer(rmContainer,
                  SchedulerUtils.createAbnormalContainerStatus(containerId,
                    SchedulerUtils.RELEASED_CONTAINER), RMContainerEventType.RELEASED);
          

          RMContainerEventType.RELEASED is not triggered by MapReduce ApplicationMaster, so we won't see this problem on MR jobs. But TEZ will triggered it when it do not need this this container, while the NodeManger will also report a container complete message to RM ,which in turn trigger the RMContainerEventType.FINISHED event. If RMContainerEventType.FINISHED event comes to RM early than TEZ AM, the problem happens.

          Show
          guoshiwei Shiwei Guo added a comment - We also seeing this problems, and it may make the RM never allocate resource for a queue that has used negative resource. I did some research and found the this is mainly caused by a race condition of calling AbstractYarnScheduler.completedContainer. Lets take FairScheduler as an example: FairSchedular.java protected synchronized void completedContainer(RMContainer rmContainer, ContainerStatus containerStatus, RMContainerEventType event) { if (rmContainer == null ) { LOG.info( "Null container completed..." ); return ; } Container container = rmContainer.getContainer(); // Get the application for the finished container FSAppAttempt application = getCurrentAttemptForContainer(container.getId()); ApplicationId appId = container.getId().getApplicationAttemptId().getApplicationId(); if (application == null ) { LOG.info( "Container " + container + " of" + " unknown application attempt " + appId + " completed with event " + event); return ; } if (!application.getLiveContainersMap().containsKey(container.getId())){ LOG.info( "Container " + container + " of application attempt " + appId + " is not alive, skip do completedContainer operation on event " + event); return ; } // Get the node on which the container was allocated FSSchedulerNode node = getFSSchedulerNode(container.getNodeId()); if (rmContainer.getState() == RMContainerState.RESERVED) { application.unreserve(rmContainer.getReservedPriority(), node); } else { application.containerCompleted(rmContainer, containerStatus, event); node.releaseContainer(container); updateRootQueueMetrics(); } LOG.info( "Application attempt " + application.getApplicationAttemptId() + " released container " + container.getId() + " on node: " + node + " with event: " + event); } completedContainer method will call application.containerCompleted, which will subtraction the resources used by this container from the usedResource counter of the application. So, if the completedContainer are called twice with the same container, the counter is subtracted too much values. So is the updateRootQueueMetrics call, so we can see negative allocatedMemory on rootQueue. The solution is to check whether the container being supplied is still live inside the completedContainer (as shown in the patch). There is some check before calling completedContainer, but that's not enough. For a more deeply discussion, the completedContainer may be called from two place: 1. Trigered by RMContainerEventType.FINISHED event: FairScheduler.nodeUpdate // Process completed containers for (ContainerStatus completedContainer : completedContainers) { ContainerId containerId = completedContainer.getContainerId(); LOG.debug( "Container FINISHED: " + containerId); completedContainer(getRMContainer(containerId), completedContainer, RMContainerEventType.FINISHED); } 2. Trigered by RMContainerEventType.RELEASED AbstractYarnScheduler.releaseContainers completedContainer(rmContainer, SchedulerUtils.createAbnormalContainerStatus(containerId, SchedulerUtils.RELEASED_CONTAINER), RMContainerEventType.RELEASED); RMContainerEventType.RELEASED is not triggered by MapReduce ApplicationMaster, so we won't see this problem on MR jobs. But TEZ will triggered it when it do not need this this container, while the NodeManger will also report a container complete message to RM ,which in turn trigger the RMContainerEventType.FINISHED event. If RMContainerEventType.FINISHED event comes to RM early than TEZ AM, the problem happens.
          Show
          guoshiwei Shiwei Guo added a comment - See this comment: https://issues.apache.org/jira/browse/YARN-3933?focusedCommentId=14709146&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14709146
          Hide
          djp Junping Du added a comment -

          I think the title here is a bit misleading. Available resource being negative shouldn't be a problem (e.g. enabling feature NM resource configuration - YARN-291) which means resource are over-commit although we shouldn't see it in most of cases. It is actually a race condition bug for FairScheduler, please mention it explicitly or developer/user may have impression that resource shouldn't be negative in any cases which we never have this assumption.

          Show
          djp Junping Du added a comment - I think the title here is a bit misleading. Available resource being negative shouldn't be a problem (e.g. enabling feature NM resource configuration - YARN-291 ) which means resource are over-commit although we shouldn't see it in most of cases. It is actually a race condition bug for FairScheduler, please mention it explicitly or developer/user may have impression that resource shouldn't be negative in any cases which we never have this assumption.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 patch 0m 1s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions.
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752004/patch.BUGFIX-JIRA-YARN-3933.txt
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / feaf034
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8896/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 patch 0m 1s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions. -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752004/patch.BUGFIX-JIRA-YARN-3933.txt Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / feaf034 Console output https://builds.apache.org/job/PreCommit-YARN-Build/8896/console This message was automatically generated.
          Hide
          guoshiwei Shiwei Guo added a comment -

          So I should better open a new issue instead?

          Show
          guoshiwei Shiwei Guo added a comment - So I should better open a new issue instead?
          Hide
          lavkesh Lavkesh Lahngir added a comment -
          Show
          lavkesh Lavkesh Lahngir added a comment - Is it related to this ? https://issues.apache.org/jira/browse/YARN-4067
          Hide
          guoshiwei Shiwei Guo added a comment -

          I think so, and so is YRAN-4045. The negative value in root queue is casued by call to updateRootQueueMetrics on same containerId. In our cluster, it has the ability to run 13000+ container, but the WEB UI says that:

          • Containers Running: -26546
          • Memory Used: -82.38 TB
          • VCores Used: -26451

          Lucky that it haven't affect scheduling yet.

          Show
          guoshiwei Shiwei Guo added a comment - I think so, and so is YRAN-4045 . The negative value in root queue is casued by call to updateRootQueueMetrics on same containerId. In our cluster, it has the ability to run 13000+ container, but the WEB UI says that: Containers Running: -26546 Memory Used: -82.38 TB VCores Used: -26451 Lucky that it haven't affect scheduling yet.
          Hide
          guoshiwei Shiwei Guo added a comment -

          I created a new YARN-4089 to describe the race condition bug for FairScheduler. I'm a newbie to the hadoop community, hope didn't do anything bad. Thanks.

          Show
          guoshiwei Shiwei Guo added a comment - I created a new YARN-4089 to describe the race condition bug for FairScheduler. I'm a newbie to the hadoop community, hope didn't do anything bad. Thanks.
          Hide
          djp Junping Du added a comment -

          Hi Shiwei Guo, we should just update the description and title for this JIRA instead of creating a new one. No worry. I will mark YARN-4089 as duplicated one for this JIRA and assign this JIRA to you given you would like to work on this and already have patch to fix it.

          Show
          djp Junping Du added a comment - Hi Shiwei Guo , we should just update the description and title for this JIRA instead of creating a new one. No worry. I will mark YARN-4089 as duplicated one for this JIRA and assign this JIRA to you given you would like to work on this and already have patch to fix it.
          Hide
          djp Junping Du added a comment -

          Looks like you are not YARN contributors yet, adding you to this elite group.

          Show
          djp Junping Du added a comment - Looks like you are not YARN contributors yet, adding you to this elite group.
          Hide
          djp Junping Du added a comment -

          This is not a lucky thing but a default behavior for YARN to schedule resource. Negative available resource just mark resource commit (consumption + reservation) is larger than current system resources which means YARN support resource over-commitment which get supported in mostly modern OS or Distributed OS. I just comment on YARN-4067 which seems to be an invalid JIRA to me.

          Show
          djp Junping Du added a comment - This is not a lucky thing but a default behavior for YARN to schedule resource. Negative available resource just mark resource commit (consumption + reservation) is larger than current system resources which means YARN support resource over-commitment which get supported in mostly modern OS or Distributed OS. I just comment on YARN-4067 which seems to be an invalid JIRA to me.
          Hide
          djp Junping Du added a comment -

          I cancel the patch, please see Jenkins' comments: "The patch file was not named according to hadoop's naming conventions". Basically, you should rename your patch with prefix as "YARN-3933" and with postfix as ".patch".

          Show
          djp Junping Du added a comment - I cancel the patch, please see Jenkins' comments: "The patch file was not named according to hadoop's naming conventions". Basically, you should rename your patch with prefix as " YARN-3933 " and with postfix as ".patch".
          Hide
          guoshiwei Shiwei Guo added a comment -

          Thanks for add me to the contributer list, so exciting!

          I have noticed Jenkins' complain and submitted a new patch in YARN-4089. And unfortunately it still not confirm to the QA standard. I'm working on it to add a unit test for this issue, and will submit it to here soon.

          Show
          guoshiwei Shiwei Guo added a comment - Thanks for add me to the contributer list, so exciting! I have noticed Jenkins' complain and submitted a new patch in YARN-4089 . And unfortunately it still not confirm to the QA standard. I'm working on it to add a unit test for this issue, and will submit it to here soon.
          Hide
          djp Junping Du added a comment -

          Sorry. I didn't see the patch you were mentioning. Did you delete it?

          Show
          djp Junping Du added a comment - Sorry. I didn't see the patch you were mentioning. Did you delete it?
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 21s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 45s There were no new javac warning messages.
          +1 javadoc 9m 48s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 50s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 27s mvn install still works.
          +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse.
          +1 findbugs 1m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 53m 40s Tests passed in hadoop-yarn-server-resourcemanager.
              92m 22s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12753166/YARN-3933.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 837fb75
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8944/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8944/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8944/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 21s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 45s There were no new javac warning messages. +1 javadoc 9m 48s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 50s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 27s mvn install still works. +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse. +1 findbugs 1m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 53m 40s Tests passed in hadoop-yarn-server-resourcemanager.     92m 22s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12753166/YARN-3933.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 837fb75 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8944/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8944/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8944/console This message was automatically generated.
          Hide
          guoshiwei Shiwei Guo added a comment -

          I improved the patch and resubmitted it here, seems OK to Hadoop QA. The old patch with bad filename is deleted by me.

          Show
          guoshiwei Shiwei Guo added a comment - I improved the patch and resubmitted it here, seems OK to Hadoop QA. The old patch with bad filename is deleted by me.
          Hide
          guoshiwei Shiwei Guo added a comment -

          Excuse me, Junping, I'm wondering what to do next about this patch, to mark it as resolved using the 'Resolve Issue' button? It works fine on our production cluster.

          Show
          guoshiwei Shiwei Guo added a comment - Excuse me, Junping, I'm wondering what to do next about this patch, to mark it as resolved using the 'Resolve Issue' button? It works fine on our production cluster.
          Hide
          guoshiwei Shiwei Guo added a comment -

          Junping Du, would you like to review the patch or give me some pointer for the next step to do ?

          Show
          guoshiwei Shiwei Guo added a comment - Junping Du , would you like to review the patch or give me some pointer for the next step to do ?
          Hide
          guoshiwei Shiwei Guo added a comment -

          Rebase on latest trunk (based on commit d0a22ba)

          Show
          guoshiwei Shiwei Guo added a comment - Rebase on latest trunk (based on commit d0a22ba)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 52s trunk passed
          +1 compile 0m 36s trunk passed with JDK v1.8.0_66
          +1 compile 0m 38s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 43s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 28s trunk passed
          +1 javadoc 0m 29s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 31s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 36s the patch passed
          +1 compile 0m 31s the patch passed with JDK v1.8.0_66
          +1 javac 0m 31s the patch passed
          +1 compile 0m 32s the patch passed with JDK v1.7.0_91
          +1 javac 0m 32s the patch passed
          +1 checkstyle 0m 17s the patch passed
          +1 mvnsite 0m 39s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 30s the patch passed
          +1 javadoc 0m 24s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 27s the patch passed with JDK v1.7.0_91
          -1 unit 61m 46s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
          -1 unit 62m 17s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          144m 34s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779909/YARN-3933.002.patch
          JIRA Issue YARN-3933
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux afde3452fd20 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ad997fa
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10120/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10120/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10120/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10120/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10120/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10120/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 52s trunk passed +1 compile 0m 36s trunk passed with JDK v1.8.0_66 +1 compile 0m 38s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 43s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 28s trunk passed +1 javadoc 0m 29s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 31s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 36s the patch passed +1 compile 0m 31s the patch passed with JDK v1.8.0_66 +1 javac 0m 31s the patch passed +1 compile 0m 32s the patch passed with JDK v1.7.0_91 +1 javac 0m 32s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 39s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 30s the patch passed +1 javadoc 0m 24s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 27s the patch passed with JDK v1.7.0_91 -1 unit 61m 46s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 62m 17s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 144m 34s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779909/YARN-3933.002.patch JIRA Issue YARN-3933 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux afde3452fd20 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ad997fa Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10120/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10120/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10120/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10120/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10120/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10120/console This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          The patch fails to apply. Shiwei Guo - mind refreshing it?

          Show
          kasha Karthik Kambatla added a comment - The patch fails to apply. Shiwei Guo - mind refreshing it?
          Hide
          guoshiwei Shiwei Guo added a comment -

          I noticed this, will fix it soon.

          Show
          guoshiwei Shiwei Guo added a comment - I noticed this, will fix it soon.
          Hide
          guoshiwei Shiwei Guo added a comment -

          Rebased onto latest trunk, and ask for code review

          Show
          guoshiwei Shiwei Guo added a comment - Rebased onto latest trunk, and ask for code review
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 50s trunk passed
          +1 compile 0m 32s trunk passed with JDK v1.8.0_74
          +1 compile 0m 32s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 0m 36s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 12s trunk passed
          +1 javadoc 0m 23s trunk passed with JDK v1.8.0_74
          +1 javadoc 0m 28s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 32s the patch passed
          +1 compile 0m 25s the patch passed with JDK v1.8.0_74
          +1 javac 0m 25s the patch passed
          +1 compile 0m 28s the patch passed with JDK v1.7.0_95
          +1 javac 0m 28s the patch passed
          +1 checkstyle 0m 16s the patch passed
          +1 mvnsite 0m 33s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 24s the patch passed
          +1 javadoc 0m 21s the patch passed with JDK v1.8.0_74
          +1 javadoc 0m 26s the patch passed with JDK v1.7.0_95
          -1 unit 68m 17s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_74.
          -1 unit 69m 2s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          155m 39s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794386/YARN-3933.003.patch
          JIRA Issue YARN-3933
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 71be9425120c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cd8b688
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10822/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10822/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10822/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10822/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10822/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10822/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 50s trunk passed +1 compile 0m 32s trunk passed with JDK v1.8.0_74 +1 compile 0m 32s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 12s trunk passed +1 javadoc 0m 23s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 28s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 32s the patch passed +1 compile 0m 25s the patch passed with JDK v1.8.0_74 +1 javac 0m 25s the patch passed +1 compile 0m 28s the patch passed with JDK v1.7.0_95 +1 javac 0m 28s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 24s the patch passed +1 javadoc 0m 21s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 26s the patch passed with JDK v1.7.0_95 -1 unit 68m 17s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_74. -1 unit 69m 2s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 155m 39s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794386/YARN-3933.003.patch JIRA Issue YARN-3933 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 71be9425120c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cd8b688 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10822/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10822/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10822/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10822/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10822/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10822/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          The fix here seems benign and should be okay to get in.

          Should we consider adding this check to SchedulerApplicationAttempt or FSAppAttempt so any other callers don't do any damage in the future?

          Show
          kasha Karthik Kambatla added a comment - The fix here seems benign and should be okay to get in. Should we consider adding this check to SchedulerApplicationAttempt or FSAppAttempt so any other callers don't do any damage in the future?
          Hide
          leftnoteasy Wangda Tan added a comment -

          Looked at this issue, it seems only FairScheduler has this issue.

          CS already checks this inside FiCaSchedulerApp. Instead of adding this separately in CS/FS, I think we can create a common completedContainer method to SchedulerApplicationAttempt. And checks liveContainers map inside the common method.

          Thoughts? Karthik Kambatla/Shiwei Guo

          Show
          leftnoteasy Wangda Tan added a comment - Looked at this issue, it seems only FairScheduler has this issue. CS already checks this inside FiCaSchedulerApp. Instead of adding this separately in CS/FS, I think we can create a common completedContainer method to SchedulerApplicationAttempt. And checks liveContainers map inside the common method. Thoughts? Karthik Kambatla / Shiwei Guo
          Hide
          sunilg Sunil G added a comment -

          As per existing patch, new liveContainers check is done before below code FS#completedContainerInternal. Pls correct me if am wrong w.r.t FS, containerCompleted need to be processed for those containers which are RESERVED too. So with current patch, this scenario may not hit.

           864     if (rmContainer.getState() == RMContainerState.RESERVED) {
           865       application.unreserve(rmContainer.getReservedPriority(), node);
           866     } else {
          
          Show
          sunilg Sunil G added a comment - As per existing patch, new liveContainers check is done before below code FS#completedContainerInternal . Pls correct me if am wrong w.r.t FS, containerCompleted need to be processed for those containers which are RESERVED too. So with current patch, this scenario may not hit. 864 if (rmContainer.getState() == RMContainerState.RESERVED) { 865 application.unreserve(rmContainer.getReservedPriority(), node); 866 } else {
          Hide
          guoshiwei Shiwei Guo added a comment -

          Sorry for long time to reply. Did you mean adding this check in the 'containerCompleted' and 'unreserve' ? I suppose that is not enough, cause the 'updateRootQueueMetrics' call in completedContainerInternal will still substract more than once the resource this container released.

          Show
          guoshiwei Shiwei Guo added a comment - Sorry for long time to reply. Did you mean adding this check in the 'containerCompleted' and 'unreserve' ? I suppose that is not enough, cause the 'updateRootQueueMetrics' call in completedContainerInternal will still substract more than once the resource this container released.
          Hide
          guoshiwei Shiwei Guo added a comment -

          Seems CS do not have this issue. And as reply to Karthik Kambatla above, the 'updateRootQueueMetrics' call requires that we must do the liveContainer check in 'completedContainerInternal' method. Share the same logic for CS/FS is good, but it may need too much refactoring for this issue? Correct me if I am wrong.

          Show
          guoshiwei Shiwei Guo added a comment - Seems CS do not have this issue. And as reply to Karthik Kambatla above, the 'updateRootQueueMetrics' call requires that we must do the liveContainer check in 'completedContainerInternal' method. Share the same logic for CS/FS is good, but it may need too much refactoring for this issue? Correct me if I am wrong.
          Hide
          guoshiwei Shiwei Guo added a comment -

          Sorry, I didn't catch the point, I am not a YARN source code expert, so may be I am lake of some background knowledge to understand your comment. Sorry.

          Show
          guoshiwei Shiwei Guo added a comment - Sorry, I didn't catch the point, I am not a YARN source code expert, so may be I am lake of some background knowledge to understand your comment. Sorry.
          Hide
          sunilg Sunil G added a comment -

          Hi

          In current patch, below code is added in FairScheduler#completedContainerInternal

           if(!application.getLiveContainersMap().containsKey(container.getId())){
          862	      LOG.info("Container " + container + " of application attempt " + appId
          863	              + " is not alive, skip do completedContainer operation on event "
          864	              + event);
          865	      return;
          866	    }
          

          So it will return if container is not present in liveContainers map. But if this container is reserved state, completedContainerInternal will return directly without invoking below code in FairScheduler#completedContainerInternal. I think this is not correct.

           864     if (rmContainer.getState() == RMContainerState.RESERVED) {
           865       application.unreserve(rmContainer.getReservedPriority(), node);
           866     } else {
          

          So I think the check to verify whether container is present in liveContainers can be done in FSAppAttempt#containerCompleted and return. CS is doing this way. So if FairScheduler opt for this, we can move this code in SchedulerApplicationAttempt. In YARN-4809, I have given a patch to make common containerCompleted for both FSAppAttempt and FiCaSchedulerApp.

          Show
          sunilg Sunil G added a comment - Hi In current patch, below code is added in FairScheduler#completedContainerInternal if (!application.getLiveContainersMap().containsKey(container.getId())){ 862 LOG.info( "Container " + container + " of application attempt " + appId 863 + " is not alive, skip do completedContainer operation on event " 864 + event); 865 return ; 866 } So it will return if container is not present in liveContainers map. But if this container is reserved state, completedContainerInternal will return directly without invoking below code in FairScheduler#completedContainerInternal . I think this is not correct. 864 if (rmContainer.getState() == RMContainerState.RESERVED) { 865 application.unreserve(rmContainer.getReservedPriority(), node); 866 } else { So I think the check to verify whether container is present in liveContainers can be done in FSAppAttempt#containerCompleted and return. CS is doing this way. So if FairScheduler opt for this, we can move this code in SchedulerApplicationAttempt . In YARN-4809 , I have given a patch to make common containerCompleted for both FSAppAttempt and FiCaSchedulerApp.
          Hide
          guoshiwei Shiwei Guo added a comment -

          Hi, I do think remove the duplicated code is a better way.
          My only concern is that the counter updated in 'FairScheduler#FaireupdateRootQueueMetrics();‘ is not protected in 'YARN-4809'. Any thoughts ?

          Show
          guoshiwei Shiwei Guo added a comment - Hi, I do think remove the duplicated code is a better way. My only concern is that the counter updated in 'FairScheduler#FaireupdateRootQueueMetrics();‘ is not protected in ' YARN-4809 '. Any thoughts ?
          Hide
          guoshiwei Shiwei Guo added a comment -

          And I didn't figure out how YARN-4809 fix the race condition, may be you mean we should apply both of the two patch?

          Show
          guoshiwei Shiwei Guo added a comment - And I didn't figure out how YARN-4809 fix the race condition, may be you mean we should apply both of the two patch?
          Hide
          sunilg Sunil G added a comment -

          YARN-4809 was only focusing on refactoring CS and FS for some common code in its App class. And its not solving this issue.

          I think the approach taken here is correct to fix the problem. But my concern was that, its better we do this check in FSAppAttempt#containerCompleted rather from FairScheduler#completedContainerInternal. So if we can check for return type from FSAppAttempt#containerCompleted, we can take a call whether to continue or return from completedContainerInternal. And I think it ll cover FaireupdateRootQueueMetrics in that case.

          And may be after fixing this, we can try merge/refactor code with YARN-4809 approach.

          Show
          sunilg Sunil G added a comment - YARN-4809 was only focusing on refactoring CS and FS for some common code in its App class. And its not solving this issue. I think the approach taken here is correct to fix the problem. But my concern was that, its better we do this check in FSAppAttempt#containerCompleted rather from FairScheduler#completedContainerInternal . So if we can check for return type from FSAppAttempt#containerCompleted , we can take a call whether to continue or return from completedContainerInternal . And I think it ll cover FaireupdateRootQueueMetrics in that case. And may be after fixing this, we can try merge/refactor code with YARN-4809 approach.
          Hide
          guoshiwei Shiwei Guo added a comment -

          got it. Thans. So let's kick YARN-3933 in Or I need to do something to let it in?

          Show
          guoshiwei Shiwei Guo added a comment - got it. Thans. So let's kick YARN-3933 in Or I need to do something to let it in?
          Hide
          sunilg Sunil G added a comment -

          Thanks. I feel this can go in independently and for other refactoring we can do in YARN-4809

          Show
          sunilg Sunil G added a comment - Thanks. I feel this can go in independently and for other refactoring we can do in YARN-4809
          Hide
          sunilg Sunil G added a comment -

          HI Shiwei Guo
          Could you pls update new patch with the suggested update.

          Show
          sunilg Sunil G added a comment - HI Shiwei Guo Could you pls update new patch with the suggested update.
          Hide
          sunilg Sunil G added a comment -

          Hi Shiwei Guo,
          Could you please help to update the patch as discussed.

          Show
          sunilg Sunil G added a comment - Hi Shiwei Guo , Could you please help to update the patch as discussed.
          Hide
          sunilg Sunil G added a comment -

          Hi Shiwei Guo
          Could you pls help to share a patch as we discussed. If you have no bandwidth, I could help to provide the same.

          Show
          sunilg Sunil G added a comment - Hi Shiwei Guo Could you pls help to share a patch as we discussed. If you have no bandwidth, I could help to provide the same.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 57s trunk passed
          +1 compile 0m 32s trunk passed
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 0m 35s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 57s trunk passed
          +1 javadoc 0m 22s trunk passed
          +1 mvninstall 0m 32s the patch passed
          +1 compile 0m 34s the patch passed
          -1 javac 0m 34s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 3 new + 4 unchanged - 0 fixed = 7 total (was 4)
          -1 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 8 new + 284 unchanged - 4 fixed = 292 total (was 288)
          +1 mvnsite 0m 34s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 8s the patch passed
          +1 javadoc 0m 18s the patch passed
          +1 unit 35m 20s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          50m 23s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e2f6409
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794386/YARN-3933.003.patch
          JIRA Issue YARN-3933
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4a6ad92f5e8d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4aefe11
          Default Java 1.8.0_91
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/12042/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12042/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12042/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12042/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 57s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 57s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 34s the patch passed -1 javac 0m 34s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 3 new + 4 unchanged - 0 fixed = 7 total (was 4) -1 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 8 new + 284 unchanged - 4 fixed = 292 total (was 288) +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 8s the patch passed +1 javadoc 0m 18s the patch passed +1 unit 35m 20s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 50m 23s Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794386/YARN-3933.003.patch JIRA Issue YARN-3933 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4a6ad92f5e8d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4aefe11 Default Java 1.8.0_91 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/12042/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12042/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12042/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12042/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          Hi Shiwei Guo
          I tried to update your patch with minor changes as we discussed earlier in this JIRA. I have not merged or moved this to a common code to share b/w Fair and CS yet. If this is fine, we can do that.

          Show
          sunilg Sunil G added a comment - Hi Shiwei Guo I tried to update your patch with minor changes as we discussed earlier in this JIRA. I have not merged or moved this to a common code to share b/w Fair and CS yet. If this is fine, we can do that.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 13s trunk passed
          +1 compile 0m 30s trunk passed
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 0m 33s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 54s trunk passed
          +1 javadoc 0m 19s trunk passed
          +1 mvninstall 0m 28s the patch passed
          +1 compile 0m 27s the patch passed
          +1 javac 0m 27s the patch passed
          -1 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 9 new + 297 unchanged - 4 fixed = 306 total (was 301)
          +1 mvnsite 0m 32s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 59s the patch passed
          +1 javadoc 0m 17s the patch passed
          -1 unit 34m 38s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          48m 17s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:85209cc
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813619/YARN-3933.004.patch
          JIRA Issue YARN-3933
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c7e1f505d547 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 73615a7
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12130/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12130/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12130/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12130/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12130/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 13s trunk passed +1 compile 0m 30s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 54s trunk passed +1 javadoc 0m 19s trunk passed +1 mvninstall 0m 28s the patch passed +1 compile 0m 27s the patch passed +1 javac 0m 27s the patch passed -1 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 9 new + 297 unchanged - 4 fixed = 306 total (was 301) +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 59s the patch passed +1 javadoc 0m 17s the patch passed -1 unit 34m 38s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 48m 17s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12813619/YARN-3933.004.patch JIRA Issue YARN-3933 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c7e1f505d547 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 73615a7 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12130/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12130/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12130/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12130/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12130/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          HI Karthik Kambatla,
          Could you pls help to review the updated patch. cc/Shiwei Guo

          Show
          sunilg Sunil G added a comment - HI Karthik Kambatla , Could you pls help to review the updated patch. cc/ Shiwei Guo
          Hide
          guoshiwei Shiwei Guo added a comment -

          Sorry, guys. I'am sorry for no reply to this issue for a long time. I currently have no bandwidth on the YARN related jobs. It's very nice to have Sunil G to kik in. Thanks very much!

          Show
          guoshiwei Shiwei Guo added a comment - Sorry, guys. I'am sorry for no reply to this issue for a long time. I currently have no bandwidth on the YARN related jobs. It's very nice to have Sunil G to kik in. Thanks very much!
          Hide
          sunilg Sunil G added a comment -

          rebased to latest trunk. pls review.

          Show
          sunilg Sunil G added a comment - rebased to latest trunk. pls review.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 39s trunk passed
          +1 compile 0m 32s trunk passed
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 0m 37s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 0m 59s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 28s the patch passed
          +1 javac 0m 28s the patch passed
          -0 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 5 new + 274 unchanged - 2 fixed = 279 total (was 276)
          +1 mvnsite 0m 35s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 1m 2s the patch passed
          +1 javadoc 0m 18s the patch passed
          -1 unit 38m 7s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          53m 12s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue YARN-3933
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835574/YARN-3933.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e732b5fc242e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b439517
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13555/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13555/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13555/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13555/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13555/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 39s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 37s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 59s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 28s the patch passed +1 javac 0m 28s the patch passed -0 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 5 new + 274 unchanged - 2 fixed = 279 total (was 276) +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 14s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 1m 2s the patch passed +1 javadoc 0m 18s the patch passed -1 unit 38m 7s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 53m 12s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-3933 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835574/YARN-3933.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e732b5fc242e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b439517 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13555/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13555/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13555/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13555/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13555/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment - - edited

          Test case failure is tracked in YARN-5043. Karthik Kambatla pls help to review the same.

          Show
          sunilg Sunil G added a comment - - edited Test case failure is tracked in YARN-5043 . Karthik Kambatla pls help to review the same.
          Hide
          kasha Karthik Kambatla added a comment -

          The approach in the patch looks reasonable. The test, however, passes me even without the code changes.

          Shiwei Guo - can you please look into it. Canceling patch in the interim.

          Show
          kasha Karthik Kambatla added a comment - The approach in the patch looks reasonable. The test, however, passes me even without the code changes. Shiwei Guo - can you please look into it. Canceling patch in the interim.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Shiwei Guo, thank you for looking into this. I ran into the same issue in YARN-6158 and I had a patch there. I attach the rebased version here for your consideration. It is very similar to the current patch, it just addresses the test issue. I am also okay with 004.patch. It would be nice to get either of the fixes checked in. What do you think?

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Shiwei Guo , thank you for looking into this. I ran into the same issue in YARN-6158 and I had a patch there. I attach the rebased version here for your consideration. It is very similar to the current patch, it just addresses the test issue. I am also okay with 004.patch. It would be nice to get either of the fixes checked in. What do you think?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 13m 21s trunk passed
          +1 compile 0m 34s trunk passed
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 0m 36s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 7s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 32s the patch passed
          +1 compile 0m 31s the patch passed
          +1 javac 0m 31s the patch passed
          +1 checkstyle 0m 21s the patch passed
          +1 mvnsite 0m 34s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 11s the patch passed
          +1 javadoc 0m 20s the patch passed
          -1 unit 59m 24s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 31s The patch does not generate ASF License warnings.
          82m 10s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation
          Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.reservation.TestFairSchedulerPlanFollower



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-3933
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12851961/YARN-3933.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 95cc0dc6953e 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 08f9397
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14874/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14874/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14874/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 21s trunk passed +1 compile 0m 34s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 7s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 11s the patch passed +1 javadoc 0m 20s the patch passed -1 unit 59m 24s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 82m 10s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.reservation.TestFairSchedulerPlanFollower Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-3933 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12851961/YARN-3933.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 95cc0dc6953e 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 08f9397 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/14874/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14874/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14874/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          I could not repro the unit test issues above.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - I could not repro the unit test issues above.
          Hide
          kasha Karthik Kambatla added a comment -

          Sorry Shiwei Guo for not getting to this sooner and thanks Miklos Szegedi for picking this up back again.

          I looked at both patches (v5 and v6); the latter looks simpler to me. The only downside is the extra updateRootQueueMetrics call, but that is likely best left for YARN-4809. +1 pending Jenkins.

          Show
          kasha Karthik Kambatla added a comment - Sorry Shiwei Guo for not getting to this sooner and thanks Miklos Szegedi for picking this up back again. I looked at both patches (v5 and v6); the latter looks simpler to me. The only downside is the extra updateRootQueueMetrics call, but that is likely best left for YARN-4809 . +1 pending Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 16m 53s trunk passed
          +1 compile 0m 31s trunk passed
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 0m 35s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 3s trunk passed
          +1 javadoc 0m 23s trunk passed
          +1 mvninstall 0m 36s the patch passed
          +1 compile 0m 35s the patch passed
          +1 javac 0m 35s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 0m 33s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 18s the patch passed
          +1 javadoc 0m 21s the patch passed
          +1 unit 44m 30s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          70m 36s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-3933
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12851961/YARN-3933.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e15d9360bc54 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 839b690
          Default Java 1.8.0_121
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14896/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14896/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 16m 53s trunk passed +1 compile 0m 31s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 3s trunk passed +1 javadoc 0m 23s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 0m 35s the patch passed +1 javac 0m 35s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 18s the patch passed +1 javadoc 0m 21s the patch passed +1 unit 44m 30s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 70m 36s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-3933 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12851961/YARN-3933.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e15d9360bc54 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 839b690 Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14896/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14896/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          Just committed this to trunk and branch-2. Thanks Shiwei Guo and Miklos Szegedi for working this, and Sunil G for the additional review.

          Show
          kasha Karthik Kambatla added a comment - Just committed this to trunk and branch-2. Thanks Shiwei Guo and Miklos Szegedi for working this, and Sunil G for the additional review.
          Hide
          kasha Karthik Kambatla added a comment -

          Also committed to branch-2.8. That had a minor conflict that I had to fix, posted the 2.8 diff here.

          Show
          kasha Karthik Kambatla added a comment - Also committed to branch-2.8. That had a minor conflict that I had to fix, posted the 2.8 diff here.
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #11240 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11240/)
          YARN-3933. FairScheduler: Multiple calls to completedContainer are not (kasha: rev 646c6d6509f515b1373288869fb92807fa2ddc9b)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #11240 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11240/ ) YARN-3933 . FairScheduler: Multiple calls to completedContainer are not (kasha: rev 646c6d6509f515b1373288869fb92807fa2ddc9b) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java

            People

            • Assignee:
              guoshiwei Shiwei Guo
              Reporter:
              lavkesh Lavkesh Lahngir
            • Votes:
              3 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development