Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4686

MiniYARNCluster.start() returns before cluster is completely started

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None

      Description

      TestRMNMInfo fails intermittently. Below is trace for the failure

      testRMNMInfo(org.apache.hadoop.mapreduce.v2.TestRMNMInfo)  Time elapsed: 0.28 sec  <<< FAILURE!
      java.lang.AssertionError: Unexpected number of live nodes: expected:<4> but was:<3>
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.junit.Assert.failNotEquals(Assert.java:743)
      	at org.junit.Assert.assertEquals(Assert.java:118)
      	at org.junit.Assert.assertEquals(Assert.java:555)
      	at org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfo(TestRMNMInfo.java:111)
      
      1. MAPREDUCE-6507.001.patch
        3 kB
        Eric Badger
      2. YARN-4686.001.patch
        7 kB
        Eric Badger
      3. YARN-4686.002.patch
        12 kB
        Eric Badger
      4. YARN-4686.003.patch
        12 kB
        Eric Badger
      5. YARN-4686.004.patch
        14 kB
        Eric Badger
      6. YARN-4686.005.patch
        15 kB
        Eric Badger
      7. YARN-4686.006.patch
        15 kB
        Eric Badger
      8. YARN-4686-branch-2.7.006.patch
        14 kB
        Eric Badger

        Issue Links

          Activity

          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Looks like MiniMRYarnCluster start is not waiting for all the NodeManagers to start and register with RM which causes random failure depends on the machine load.

          Show
          rohithsharma Rohith Sharma K S added a comment - Looks like MiniMRYarnCluster start is not waiting for all the NodeManagers to start and register with RM which causes random failure depends on the machine load.
          Hide
          ebadger Eric Badger added a comment -

          The problem is that there is a thread starting the services and the main thread is asynchronously checking the state. Thus, the main thread checks that the services are fully up, but they can appear to be started before they have actually completed their startup (AbstractService sets the state to STARTED before calling serviceStart()). I think that the easiest fix is to remove the threads that are spawned from within the main thread that do not appear to accomplish anything of value.

          Show
          ebadger Eric Badger added a comment - The problem is that there is a thread starting the services and the main thread is asynchronously checking the state. Thus, the main thread checks that the services are fully up, but they can appear to be started before they have actually completed their startup (AbstractService sets the state to STARTED before calling serviceStart()). I think that the easiest fix is to remove the threads that are spawned from within the main thread that do not appear to accomplish anything of value.
          Hide
          ebadger Eric Badger added a comment -

          I noticed that TestMRTimelineEventHandling is failing in branch-2.7. I believe that this patch will also fix this failing test.

          Show
          ebadger Eric Badger added a comment - I noticed that TestMRTimelineEventHandling is failing in branch-2.7. I believe that this patch will also fix this failing test.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 10m 53s trunk passed
          +1 compile 0m 23s trunk passed with JDK v1.8.0_66
          +1 compile 0m 21s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 28s trunk passed
          +1 mvneclipse 0m 20s trunk passed
          +1 findbugs 0m 36s trunk passed
          +1 javadoc 0m 27s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 22s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 0m 28s the patch passed
          +1 compile 0m 25s the patch passed with JDK v1.8.0_66
          +1 javac 0m 25s the patch passed
          +1 compile 0m 21s the patch passed with JDK v1.7.0_91
          +1 javac 0m 21s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 0m 27s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 51s the patch passed
          +1 javadoc 0m 26s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 20s the patch passed with JDK v1.7.0_91
          -1 unit 7m 40s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_66.
          -1 unit 7m 25s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 27s Patch does not generate ASF License warnings.
          35m 17s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.TestMiniYARNClusterForHA
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.TestMiniYARNClusterForHA
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785871/MAPREDUCE-6507.001.patch
          JIRA Issue MAPREDUCE-6507
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9ac25ba8cbdc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6d12138
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6306/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6306/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6306/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6306/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6306/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6306/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 10m 53s trunk passed +1 compile 0m 23s trunk passed with JDK v1.8.0_66 +1 compile 0m 21s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 0m 36s trunk passed +1 javadoc 0m 27s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 22s trunk passed with JDK v1.7.0_91 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 0m 28s the patch passed +1 compile 0m 25s the patch passed with JDK v1.8.0_66 +1 javac 0m 25s the patch passed +1 compile 0m 21s the patch passed with JDK v1.7.0_91 +1 javac 0m 21s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 27s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 51s the patch passed +1 javadoc 0m 26s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 20s the patch passed with JDK v1.7.0_91 -1 unit 7m 40s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_66. -1 unit 7m 25s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 35m 17s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.TestMiniYARNClusterForHA   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.TestMiniYARNClusterForHA   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785871/MAPREDUCE-6507.001.patch JIRA Issue MAPREDUCE-6507 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9ac25ba8cbdc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6d12138 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6306/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6306/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6306/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6306/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6306/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6306/console This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Tests are failing because of a race condition between the RM startup and the NM startup. In each of their serviceStart() methods, they are spawning new threads to call start(), which introduces the race. The NM is set up with a waitCount of up to 60 seconds, so that it can wait for the cluster to complete startup (even though the start method for the RM has already returned). Removing the threads fixes the race in the test that prompted this Jira (TestRMNMInfo), but causes other tests to fail. Any tests that start up the MiniYARNCluster cluster without an active RM will fail because the node managers block the main thread from transitioning one of the RMs from standby to active. This is why the threads worked, since it allowed the NMs to wait, while the main thread zoomed by and transitioned a standby RM to active.

          I propose changing the MiniYARNCluster start method such that it does not complete until the cluster is completely started and to always make one RM active in HA setups. This will require changes to the affected tests (TestRMFailover, TestMiniYARNClusterForHA, etc.), but makes the code more understandable and removes races. The tests are only passing right now because of excessive timeouts to mask the race that they're fighting.

          Karthik Kambatla Jason Lowe Please advise.

          Show
          ebadger Eric Badger added a comment - Tests are failing because of a race condition between the RM startup and the NM startup. In each of their serviceStart() methods, they are spawning new threads to call start(), which introduces the race. The NM is set up with a waitCount of up to 60 seconds, so that it can wait for the cluster to complete startup (even though the start method for the RM has already returned). Removing the threads fixes the race in the test that prompted this Jira (TestRMNMInfo), but causes other tests to fail. Any tests that start up the MiniYARNCluster cluster without an active RM will fail because the node managers block the main thread from transitioning one of the RMs from standby to active. This is why the threads worked, since it allowed the NMs to wait, while the main thread zoomed by and transitioned a standby RM to active. I propose changing the MiniYARNCluster start method such that it does not complete until the cluster is completely started and to always make one RM active in HA setups. This will require changes to the affected tests (TestRMFailover, TestMiniYARNClusterForHA, etc.), but makes the code more understandable and removes races. The tests are only passing right now because of excessive timeouts to mask the race that they're fighting. Karthik Kambatla Jason Lowe Please advise.
          Hide
          ebadger Eric Badger added a comment -

          Moving to YARN project, since this affects the MiniYarnCluster.

          Show
          ebadger Eric Badger added a comment - Moving to YARN project, since this affects the MiniYarnCluster.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 11m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 1s trunk passed
          +1 compile 0m 16s trunk passed with JDK v1.8.0_72
          +1 compile 0m 17s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 23s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 0m 28s trunk passed
          +1 javadoc 0m 16s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 18s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 17s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.8.0_72
          +1 javac 0m 14s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.7.0_95
          +1 javac 0m 14s the patch passed
          +1 checkstyle 0m 10s the patch passed
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 41s the patch passed
          +1 javadoc 0m 14s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 17s the patch passed with JDK v1.7.0_95
          -1 unit 6m 44s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_72.
          -1 unit 6m 39s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          39m 10s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.TestMiniYARNClusterForHA
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.server.TestContainerManagerSecurity
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.TestMiniYARNClusterForHA
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.server.TestContainerManagerSecurity



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785871/MAPREDUCE-6507.001.patch
          JIRA Issue YARN-4686
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux bf4e47bcac29 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cb3a103
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10548/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10548/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10548/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10548/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10548/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
          Max memory used 77MB
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10548/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 1s trunk passed +1 compile 0m 16s trunk passed with JDK v1.8.0_72 +1 compile 0m 17s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 16s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 18s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 17s the patch passed +1 compile 0m 14s the patch passed with JDK v1.8.0_72 +1 javac 0m 14s the patch passed +1 compile 0m 14s the patch passed with JDK v1.7.0_95 +1 javac 0m 14s the patch passed +1 checkstyle 0m 10s the patch passed +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 41s the patch passed +1 javadoc 0m 14s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 17s the patch passed with JDK v1.7.0_95 -1 unit 6m 44s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_72. -1 unit 6m 39s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 39m 10s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.TestMiniYARNClusterForHA   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestContainerManagerSecurity JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.TestMiniYARNClusterForHA   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestContainerManagerSecurity Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785871/MAPREDUCE-6507.001.patch JIRA Issue YARN-4686 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bf4e47bcac29 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cb3a103 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10548/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10548/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10548/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10548/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10548/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-YARN-Build/10548/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          I'd really like to see the minicluster not startup by default with a race condition where it hasn't actually finished starting. With multiple tests currently failing sporadically due to this, I'd like the start() method to not return until the cluster is started. For non-HA setups this seems very straightforward.

          However for the HA minicluster it appears the intent is to have the RMs all come up in standby. The problem is that the NM start method will not return until it has successfully registered with an RM. Since all RMs are in standby the NM start never completes, the minicluster start never completes, and we never get to the part of the test where it activates an RM. Therefore HA minicluster tests will always timeout.

          I like Eric's proposal to have the minicluster activate the first RM during the start method of an HA cluster so we can bring it up and return from the cluster start method with no pending start processing (and therefore race conditions in the test using the cluster). However that could break some of the assumptions of those using the HA minicluster in their existing tests. For Hadoop tests we can simply fixup the tests accordingly, if necessary (since most seem to activate the first one anyway), but I don't know if there are other tests that use an HA minicluster and will break if the first RM is already active by default.

          Karthik Kambatla do you have an opinion on this?

          Show
          jlowe Jason Lowe added a comment - I'd really like to see the minicluster not startup by default with a race condition where it hasn't actually finished starting. With multiple tests currently failing sporadically due to this, I'd like the start() method to not return until the cluster is started. For non-HA setups this seems very straightforward. However for the HA minicluster it appears the intent is to have the RMs all come up in standby. The problem is that the NM start method will not return until it has successfully registered with an RM. Since all RMs are in standby the NM start never completes, the minicluster start never completes, and we never get to the part of the test where it activates an RM. Therefore HA minicluster tests will always timeout. I like Eric's proposal to have the minicluster activate the first RM during the start method of an HA cluster so we can bring it up and return from the cluster start method with no pending start processing (and therefore race conditions in the test using the cluster). However that could break some of the assumptions of those using the HA minicluster in their existing tests. For Hadoop tests we can simply fixup the tests accordingly, if necessary (since most seem to activate the first one anyway), but I don't know if there are other tests that use an HA minicluster and will break if the first RM is already active by default. Karthik Kambatla do you have an opinion on this?
          Hide
          kasha Karthik Kambatla added a comment -

          It makes sense to ensure the mini-cluster start doesn't return until the cluster has actually fully started. I am comfortable with transitioning one of the RMs to active and updating the HA tests accordingly.

          I don't expect tests outside of Yarn/MR to depend on HA nature of the cluster. Also, MiniYarnCluster is not marked Public-Stable yet. Should we just go ahead and mark the constructor that allows multiple RMs Private?

          Show
          kasha Karthik Kambatla added a comment - It makes sense to ensure the mini-cluster start doesn't return until the cluster has actually fully started. I am comfortable with transitioning one of the RMs to active and updating the HA tests accordingly. I don't expect tests outside of Yarn/MR to depend on HA nature of the cluster. Also, MiniYarnCluster is not marked Public-Stable yet. Should we just go ahead and mark the constructor that allows multiple RMs Private?
          Hide
          ebadger Eric Badger added a comment -

          Transition 1 RM to active in MiniYarnCluster start. Also improve MiniYarnCluster start by waiting for Scheduler to see all NMs.

          Show
          ebadger Eric Badger added a comment - Transition 1 RM to active in MiniYarnCluster start. Also improve MiniYarnCluster start by waiting for Scheduler to see all NMs.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 7m 4s trunk passed
          +1 compile 2m 11s trunk passed with JDK v1.8.0_72
          +1 compile 2m 14s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 0m 47s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 1m 5s trunk passed
          +1 javadoc 0m 31s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 35s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 0m 36s the patch passed
          +1 compile 2m 8s the patch passed with JDK v1.8.0_72
          +1 javac 2m 8s the patch passed
          +1 compile 2m 15s the patch passed with JDK v1.7.0_95
          +1 javac 2m 15s the patch passed
          +1 checkstyle 0m 30s the patch passed
          +1 mvnsite 0m 42s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 23s the patch passed
          +1 javadoc 0m 24s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 31s the patch passed with JDK v1.7.0_95
          -1 unit 6m 3s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_72.
          -1 unit 64m 15s hadoop-yarn-client in the patch failed with JDK v1.8.0_72.
          -1 unit 6m 6s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95.
          -1 unit 64m 8s hadoop-yarn-client in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          167m 2s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.client.TestGetGroups
            hadoop.yarn.client.TestApplicationClientProtocolOnHA
          JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI
            org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
            org.apache.hadoop.yarn.client.api.impl.TestYarnClient
            org.apache.hadoop.yarn.client.api.impl.TestNMClient
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.client.TestGetGroups
            hadoop.yarn.client.TestApplicationClientProtocolOnHA
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI
            org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
            org.apache.hadoop.yarn.client.api.impl.TestYarnClient
            org.apache.hadoop.yarn.client.api.impl.TestNMClient



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790026/YARN-4686.001.patch
          JIRA Issue YARN-4686
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7c15e70e51a2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b2951f9
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10642/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10642/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 7m 4s trunk passed +1 compile 2m 11s trunk passed with JDK v1.8.0_72 +1 compile 2m 14s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 47s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 1m 5s trunk passed +1 javadoc 0m 31s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 35s trunk passed with JDK v1.7.0_95 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 36s the patch passed +1 compile 2m 8s the patch passed with JDK v1.8.0_72 +1 javac 2m 8s the patch passed +1 compile 2m 15s the patch passed with JDK v1.7.0_95 +1 javac 2m 15s the patch passed +1 checkstyle 0m 30s the patch passed +1 mvnsite 0m 42s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 23s the patch passed +1 javadoc 0m 24s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 31s the patch passed with JDK v1.7.0_95 -1 unit 6m 3s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_72. -1 unit 64m 15s hadoop-yarn-client in the patch failed with JDK v1.8.0_72. -1 unit 6m 6s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95. -1 unit 64m 8s hadoop-yarn-client in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 167m 2s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.client.TestGetGroups   hadoop.yarn.client.TestApplicationClientProtocolOnHA JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient   org.apache.hadoop.yarn.client.api.impl.TestYarnClient   org.apache.hadoop.yarn.client.api.impl.TestNMClient JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.client.TestGetGroups   hadoop.yarn.client.TestApplicationClientProtocolOnHA JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient   org.apache.hadoop.yarn.client.api.impl.TestYarnClient   org.apache.hadoop.yarn.client.api.impl.TestNMClient Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790026/YARN-4686.001.patch JIRA Issue YARN-4686 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7c15e70e51a2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b2951f9 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10642/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10642/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/10642/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          I am unable to reproduce any of the JUnit timeout failures (TestYarnCLI, TestAMRMClient, TestYarnClient, TestNMClient) locally via trunk or via trunk with my patch added.

          TestMiniYarnClusterNodeUtilization fails locally in both trunk and with my patch.

          TestContainerManagerSecurity passes locally in both trunk and with my patch.

          TestGetGroups has an Ignore annotation, so we can probably ignore that error.

          TestApplicationClientProtocolOnHA passes locally on trunk and fails locally with my patch. However, there are a non-deterministic amount of tests failing with the same error in the initialization code.

          java.lang.AssertionError: NMs failed to connect to the RM
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.assertTrue(Assert.java:41)
          	at org.apache.hadoop.yarn.client.ProtocolHATestBase.verifyConnections(ProtocolHATestBase.java:219)
          	at org.apache.hadoop.yarn.client.ProtocolHATestBase.startHACluster(ProtocolHATestBase.java:284)
          	at org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA.initiate(TestApplicationClientProtocolOnHA.java:54)
          
          Show
          ebadger Eric Badger added a comment - I am unable to reproduce any of the JUnit timeout failures (TestYarnCLI, TestAMRMClient, TestYarnClient, TestNMClient) locally via trunk or via trunk with my patch added. TestMiniYarnClusterNodeUtilization fails locally in both trunk and with my patch. TestContainerManagerSecurity passes locally in both trunk and with my patch. TestGetGroups has an Ignore annotation, so we can probably ignore that error. TestApplicationClientProtocolOnHA passes locally on trunk and fails locally with my patch. However, there are a non-deterministic amount of tests failing with the same error in the initialization code. java.lang.AssertionError: NMs failed to connect to the RM at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.yarn.client.ProtocolHATestBase.verifyConnections(ProtocolHATestBase.java:219) at org.apache.hadoop.yarn.client.ProtocolHATestBase.startHACluster(ProtocolHATestBase.java:284) at org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA.initiate(TestApplicationClientProtocolOnHA.java:54)
          Hide
          ebadger Eric Badger added a comment -

          TestYarnCLI, TestAMRMClient, TestYarnClient, TestNMClient, and TestGetGroups are failing in multiple recent precommit builds YARN-4117, YARN-4630, YARN-4676.

          TestMiniYarnClusterNodeUtilization is tracked by YARN-4566.

          TestContainerManagerSecurity is failing on other recent precommit builds YARN-4117, YARN-4566.

          The only failure related to this patch is TestApplicationClientProtocolOnHA.

          Show
          ebadger Eric Badger added a comment - TestYarnCLI, TestAMRMClient, TestYarnClient, TestNMClient, and TestGetGroups are failing in multiple recent precommit builds YARN-4117 , YARN-4630 , YARN-4676 . TestMiniYarnClusterNodeUtilization is tracked by YARN-4566 . TestContainerManagerSecurity is failing on other recent precommit builds YARN-4117 , YARN-4566 . The only failure related to this patch is TestApplicationClientProtocolOnHA.
          Hide
          ebadger Eric Badger added a comment -

          This patch fixes a race issue between the NMs resyncing with the RM and the NMs stopping via serviceStop.

          Show
          ebadger Eric Badger added a comment - This patch fixes a race issue between the NMs resyncing with the RM and the NMs stopping via serviceStop.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 12s Maven dependency ordering for branch
          +1 mvninstall 6m 39s trunk passed
          +1 compile 1m 42s trunk passed with JDK v1.8.0_72
          +1 compile 2m 4s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 1m 13s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 0m 45s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 56s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 0m 58s the patch passed
          +1 compile 1m 44s the patch passed with JDK v1.8.0_72
          +1 javac 1m 44s the patch passed
          +1 compile 2m 3s the patch passed with JDK v1.7.0_95
          +1 javac 2m 3s the patch passed
          -1 checkstyle 0m 32s hadoop-yarn-project/hadoop-yarn: patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30)
          +1 mvnsite 1m 8s the patch passed
          +1 mvneclipse 0m 37s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 20s the patch passed
          +1 javadoc 0m 40s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 51s the patch passed with JDK v1.7.0_95
          -1 unit 8m 55s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_72.
          -1 unit 6m 44s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_72.
          -1 unit 63m 12s hadoop-yarn-client in the patch failed with JDK v1.8.0_72.
          -1 unit 9m 22s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_95.
          -1 unit 6m 57s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95.
          -1 unit 63m 31s hadoop-yarn-client in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          188m 19s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.nodemanager.TestNodeManagerResync
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.client.TestGetGroups
          JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI
            org.apache.hadoop.yarn.client.api.impl.TestYarnClient
            org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
            org.apache.hadoop.yarn.client.api.impl.TestNMClient
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.nodemanager.TestNodeManagerResync
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.client.TestGetGroups
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI
            org.apache.hadoop.yarn.client.api.impl.TestYarnClient
            org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
            org.apache.hadoop.yarn.client.api.impl.TestNMClient



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791067/YARN-4686.002.patch
          JIRA Issue YARN-4686
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 54af564fda6d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 67880cc
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10692/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10692/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 6m 39s trunk passed +1 compile 1m 42s trunk passed with JDK v1.8.0_72 +1 compile 2m 4s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 13s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 45s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 56s trunk passed with JDK v1.7.0_95 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 58s the patch passed +1 compile 1m 44s the patch passed with JDK v1.8.0_72 +1 javac 1m 44s the patch passed +1 compile 2m 3s the patch passed with JDK v1.7.0_95 +1 javac 2m 3s the patch passed -1 checkstyle 0m 32s hadoop-yarn-project/hadoop-yarn: patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30) +1 mvnsite 1m 8s the patch passed +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 20s the patch passed +1 javadoc 0m 40s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 51s the patch passed with JDK v1.7.0_95 -1 unit 8m 55s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_72. -1 unit 6m 44s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_72. -1 unit 63m 12s hadoop-yarn-client in the patch failed with JDK v1.8.0_72. -1 unit 9m 22s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_95. -1 unit 6m 57s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95. -1 unit 63m 31s hadoop-yarn-client in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 188m 19s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.nodemanager.TestNodeManagerResync   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.client.TestGetGroups JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI   org.apache.hadoop.yarn.client.api.impl.TestYarnClient   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient   org.apache.hadoop.yarn.client.api.impl.TestNMClient JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.nodemanager.TestNodeManagerResync   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.client.TestGetGroups JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI   org.apache.hadoop.yarn.client.api.impl.TestYarnClient   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient   org.apache.hadoop.yarn.client.api.impl.TestNMClient Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791067/YARN-4686.002.patch JIRA Issue YARN-4686 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 54af564fda6d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 67880cc Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10692/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/10692/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Fixing a deadlock issue related to the Node Status Updater thread and the Reboot thread. Also taking out heartbeatMonitor notify optimization. I will file a separate JIRA for this issue as well as the test whose race condition it exposes.

          Show
          ebadger Eric Badger added a comment - Fixing a deadlock issue related to the Node Status Updater thread and the Reboot thread. Also taking out heartbeatMonitor notify optimization. I will file a separate JIRA for this issue as well as the test whose race condition it exposes.
          Hide
          ebadger Eric Badger added a comment -

          JIRA YARN-4756 has been opened regarding the heartbeatMonitor optimization in the startStatusUpdater thread.

          Show
          ebadger Eric Badger added a comment - JIRA YARN-4756 has been opened regarding the heartbeatMonitor optimization in the startStatusUpdater thread.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 6m 40s trunk passed
          +1 compile 1m 44s trunk passed with JDK v1.8.0_74
          +1 compile 2m 3s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 11s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 1m 49s trunk passed
          +1 javadoc 0m 43s trunk passed with JDK v1.8.0_74
          +1 javadoc 0m 54s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 58s the patch passed
          +1 compile 1m 42s the patch passed with JDK v1.8.0_74
          +1 javac 1m 42s the patch passed
          +1 compile 2m 2s the patch passed with JDK v1.7.0_95
          +1 javac 2m 2s the patch passed
          -1 checkstyle 0m 29s hadoop-yarn-project/hadoop-yarn: patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30)
          +1 mvnsite 1m 6s the patch passed
          +1 mvneclipse 0m 34s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 19s the patch passed
          +1 javadoc 0m 40s the patch passed with JDK v1.8.0_74
          +1 javadoc 0m 50s the patch passed with JDK v1.7.0_95
          +1 unit 9m 5s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74.
          -1 unit 6m 20s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_74.
          -1 unit 63m 43s hadoop-yarn-client in the patch failed with JDK v1.8.0_74.
          +1 unit 9m 29s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
          -1 unit 6m 31s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95.
          -1 unit 63m 58s hadoop-yarn-client in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          188m 9s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.client.TestGetGroups
          JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI
            org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
            org.apache.hadoop.yarn.client.api.impl.TestYarnClient
            org.apache.hadoop.yarn.client.api.impl.TestNMClient
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.client.TestGetGroups
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI
            org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
            org.apache.hadoop.yarn.client.api.impl.TestYarnClient
            org.apache.hadoop.yarn.client.api.impl.TestNMClient



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791207/YARN-4686.003.patch
          JIRA Issue YARN-4686
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 28cc5fd95deb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0a9f00a
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10698/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10698/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 6m 40s trunk passed +1 compile 1m 44s trunk passed with JDK v1.8.0_74 +1 compile 2m 3s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 11s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 1m 49s trunk passed +1 javadoc 0m 43s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 54s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 58s the patch passed +1 compile 1m 42s the patch passed with JDK v1.8.0_74 +1 javac 1m 42s the patch passed +1 compile 2m 2s the patch passed with JDK v1.7.0_95 +1 javac 2m 2s the patch passed -1 checkstyle 0m 29s hadoop-yarn-project/hadoop-yarn: patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30) +1 mvnsite 1m 6s the patch passed +1 mvneclipse 0m 34s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 19s the patch passed +1 javadoc 0m 40s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 50s the patch passed with JDK v1.7.0_95 +1 unit 9m 5s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74. -1 unit 6m 20s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_74. -1 unit 63m 43s hadoop-yarn-client in the patch failed with JDK v1.8.0_74. +1 unit 9m 29s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. -1 unit 6m 31s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95. -1 unit 63m 58s hadoop-yarn-client in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 188m 9s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.client.TestGetGroups JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient   org.apache.hadoop.yarn.client.api.impl.TestYarnClient   org.apache.hadoop.yarn.client.api.impl.TestNMClient JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.client.TestGetGroups JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient   org.apache.hadoop.yarn.client.api.impl.TestYarnClient   org.apache.hadoop.yarn.client.api.impl.TestNMClient Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791207/YARN-4686.003.patch JIRA Issue YARN-4686 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 28cc5fd95deb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0a9f00a Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10698/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10698/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/10698/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          As per above comment, these test failures are not related to the patch. All relevant test failures have been addressed. Jason Lowe Karthik Kambatla Please review the patch when you get a chance.

          Show
          ebadger Eric Badger added a comment - As per above comment, these test failures are not related to the patch. All relevant test failures have been addressed. Jason Lowe Karthik Kambatla Please review the patch when you get a chance.
          Hide
          eepayne Eric Payne added a comment -

          Eric Badger, Thanks for working on a fix this issue.

          The patch is causing a failure with TestYarnClient:

          TestYarnClient.testReservationAPIs:1209 The size of the largest gang in the reservation refinition (<memory:1024, vCores:1>) exceed the capacity available (<memory:0, vCores:0> )
          

          Other than that, the patch looks good to me. I am a little concerned about the removal of extra threads in MiniYARNCluster#startResourceManager and MiniYARNCluster#*#serviceStart. I would be interested in Jason Lowe's and Karthik Kambatla's thoughts.

          Show
          eepayne Eric Payne added a comment - Eric Badger , Thanks for working on a fix this issue. The patch is causing a failure with TestYarnClient : TestYarnClient.testReservationAPIs:1209 The size of the largest gang in the reservation refinition (<memory:1024, vCores:1>) exceed the capacity available (<memory:0, vCores:0> ) Other than that, the patch looks good to me. I am a little concerned about the removal of extra threads in MiniYARNCluster#startResourceManager and MiniYARNCluster#*#serviceStart . I would be interested in Jason Lowe 's and Karthik Kambatla 's thoughts.
          Hide
          ebadger Eric Badger added a comment -

          Thanks for finding this failure, Eric Payne.

          I narrowed down the issue to being a race condition between the MiniYARNCluster being completely started and the reservation being placed via the test. When the CapacityScheduler starts up, it creates a a PlanFollower (via startPlanFollower()). The thread created by startPlanFollower() executes the synchronizePlan() function in a loop. The main test code in TestYarnClient#testReservationAPIs is running in a different thread and calls submitReservation (TetsYarnClient.java:1213) once the cluster is up and running. The race is between the synchronizePlan thread calling plan.setTotalCapacity (indirectly through CapacityScheduler.java:137) and the submitReservation thread calling plan.getTotalCapacity (indirectly through ReservationInputValidator.java:148).

          The patch that I submitted before makes sure that the MiniYARNCluster won't return until the CapacityScheduler has registered all of the nodes, but it doesn't wait for the totalCapacity to be set to the correct value. Is there a good way to make sure that the cluster won't start until the scheduler has totalCapacity set to the correct value?

          Show
          ebadger Eric Badger added a comment - Thanks for finding this failure, Eric Payne . I narrowed down the issue to being a race condition between the MiniYARNCluster being completely started and the reservation being placed via the test. When the CapacityScheduler starts up, it creates a a PlanFollower (via startPlanFollower()). The thread created by startPlanFollower() executes the synchronizePlan() function in a loop. The main test code in TestYarnClient#testReservationAPIs is running in a different thread and calls submitReservation (TetsYarnClient.java:1213) once the cluster is up and running. The race is between the synchronizePlan thread calling plan.setTotalCapacity (indirectly through CapacityScheduler.java:137) and the submitReservation thread calling plan.getTotalCapacity (indirectly through ReservationInputValidator.java:148). The patch that I submitted before makes sure that the MiniYARNCluster won't return until the CapacityScheduler has registered all of the nodes, but it doesn't wait for the totalCapacity to be set to the correct value. Is there a good way to make sure that the cluster won't start until the scheduler has totalCapacity set to the correct value?
          Hide
          ebadger Eric Badger added a comment -

          The new patch changes the waitForNodeManagersToConnect method in MiniYARNCluster.java so that it waits for the total plan capacity to be greater than 0 (to ensure that reservations can be made). It fixes the TestYarnClient#testReservationAPIs test failure locally on my machine. However, I'm not sure if this check should be in the MiniYARNCluster itself or whether it should be in the test code that calls it, since only a small amount of tests will actually be worried about the reservation system.

          Show
          ebadger Eric Badger added a comment - The new patch changes the waitForNodeManagersToConnect method in MiniYARNCluster.java so that it waits for the total plan capacity to be greater than 0 (to ensure that reservations can be made). It fixes the TestYarnClient#testReservationAPIs test failure locally on my machine. However, I'm not sure if this check should be in the MiniYARNCluster itself or whether it should be in the test code that calls it, since only a small amount of tests will actually be worried about the reservation system.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 21s Maven dependency ordering for branch
          +1 mvninstall 6m 30s trunk passed
          +1 compile 1m 43s trunk passed with JDK v1.8.0_74
          +1 compile 2m 3s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 1m 9s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 1m 49s trunk passed
          +1 javadoc 0m 45s trunk passed with JDK v1.8.0_74
          +1 javadoc 0m 55s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 0m 58s the patch passed
          +1 compile 1m 40s the patch passed with JDK v1.8.0_74
          +1 javac 1m 40s the patch passed
          +1 compile 2m 2s the patch passed with JDK v1.7.0_95
          +1 javac 2m 2s the patch passed
          -1 checkstyle 0m 30s hadoop-yarn-project/hadoop-yarn: patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30)
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 35s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 19s the patch passed
          +1 javadoc 0m 39s the patch passed with JDK v1.8.0_74
          +1 javadoc 0m 49s the patch passed with JDK v1.7.0_95
          +1 unit 9m 5s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74.
          -1 unit 1m 14s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_74.
          -1 unit 17m 31s hadoop-yarn-client in the patch failed with JDK v1.8.0_74.
          +1 unit 9m 37s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
          -1 unit 1m 21s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95.
          -1 unit 17m 51s hadoop-yarn-client in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          85m 47s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.TestMiniYARNClusterForHA
            hadoop.yarn.server.TestDiskFailures
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnCluster
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.client.TestResourceTrackerOnHA
            hadoop.yarn.client.api.impl.TestAMRMClient
            hadoop.yarn.client.TestGetGroups
            hadoop.yarn.client.api.impl.TestNMClient
            hadoop.yarn.client.TestHedgingRequestRMFailoverProxyProvider
            hadoop.yarn.client.TestApplicationClientProtocolOnHA
            hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA
            hadoop.yarn.client.TestRMFailover
            hadoop.yarn.client.cli.TestYarnCLI
          JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.yarn.client.api.impl.TestYarnClient
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.TestMiniYARNClusterForHA
            hadoop.yarn.server.TestDiskFailures
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnCluster
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.client.TestResourceTrackerOnHA
            hadoop.yarn.client.api.impl.TestAMRMClient
            hadoop.yarn.client.TestGetGroups
            hadoop.yarn.client.api.impl.TestNMClient
            hadoop.yarn.client.TestHedgingRequestRMFailoverProxyProvider
            hadoop.yarn.client.TestApplicationClientProtocolOnHA
            hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA
            hadoop.yarn.client.TestRMFailover
            hadoop.yarn.client.cli.TestYarnCLI
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.yarn.client.api.impl.TestYarnClient



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793312/YARN-4686.004.patch
          JIRA Issue YARN-4686
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cd715f72ec09 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 658ee95
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10776/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10776/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 21s Maven dependency ordering for branch +1 mvninstall 6m 30s trunk passed +1 compile 1m 43s trunk passed with JDK v1.8.0_74 +1 compile 2m 3s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 9s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 1m 49s trunk passed +1 javadoc 0m 45s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 55s trunk passed with JDK v1.7.0_95 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 0m 58s the patch passed +1 compile 1m 40s the patch passed with JDK v1.8.0_74 +1 javac 1m 40s the patch passed +1 compile 2m 2s the patch passed with JDK v1.7.0_95 +1 javac 2m 2s the patch passed -1 checkstyle 0m 30s hadoop-yarn-project/hadoop-yarn: patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30) +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 19s the patch passed +1 javadoc 0m 39s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 49s the patch passed with JDK v1.7.0_95 +1 unit 9m 5s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74. -1 unit 1m 14s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_74. -1 unit 17m 31s hadoop-yarn-client in the patch failed with JDK v1.8.0_74. +1 unit 9m 37s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. -1 unit 1m 21s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95. -1 unit 17m 51s hadoop-yarn-client in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 85m 47s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.TestMiniYARNClusterForHA   hadoop.yarn.server.TestDiskFailures   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnCluster   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.client.TestResourceTrackerOnHA   hadoop.yarn.client.api.impl.TestAMRMClient   hadoop.yarn.client.TestGetGroups   hadoop.yarn.client.api.impl.TestNMClient   hadoop.yarn.client.TestHedgingRequestRMFailoverProxyProvider   hadoop.yarn.client.TestApplicationClientProtocolOnHA   hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA   hadoop.yarn.client.TestRMFailover   hadoop.yarn.client.cli.TestYarnCLI JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.yarn.client.api.impl.TestYarnClient JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.TestMiniYARNClusterForHA   hadoop.yarn.server.TestDiskFailures   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnCluster   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.client.TestResourceTrackerOnHA   hadoop.yarn.client.api.impl.TestAMRMClient   hadoop.yarn.client.TestGetGroups   hadoop.yarn.client.api.impl.TestNMClient   hadoop.yarn.client.TestHedgingRequestRMFailoverProxyProvider   hadoop.yarn.client.TestApplicationClientProtocolOnHA   hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA   hadoop.yarn.client.TestRMFailover   hadoop.yarn.client.cli.TestYarnCLI JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.yarn.client.api.impl.TestYarnClient Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793312/YARN-4686.004.patch JIRA Issue YARN-4686 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cd715f72ec09 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 658ee95 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10776/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10776/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/10776/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          I moved the total capacity check out of the waitForNodeManagersToConnect method and into the TestYarnClient#testReservationAPIs test so as to not fail other tests.

          Show
          ebadger Eric Badger added a comment - I moved the total capacity check out of the waitForNodeManagersToConnect method and into the TestYarnClient#testReservationAPIs test so as to not fail other tests.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 7m 49s trunk passed
          +1 compile 2m 7s trunk passed with JDK v1.8.0_74
          +1 compile 2m 28s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 39s trunk passed
          +1 mvnsite 1m 27s trunk passed
          +1 mvneclipse 0m 44s trunk passed
          +1 findbugs 2m 0s trunk passed
          +1 javadoc 0m 51s trunk passed with JDK v1.8.0_74
          +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 1m 9s the patch passed
          +1 compile 2m 9s the patch passed with JDK v1.8.0_74
          +1 javac 2m 9s the patch passed
          +1 compile 2m 23s the patch passed with JDK v1.7.0_95
          +1 javac 2m 23s the patch passed
          -1 checkstyle 0m 35s hadoop-yarn-project/hadoop-yarn: patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30)
          +1 mvnsite 1m 12s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 33s the patch passed
          +1 javadoc 0m 53s the patch passed with JDK v1.8.0_74
          +1 javadoc 0m 54s the patch passed with JDK v1.7.0_95
          +1 unit 9m 57s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74.
          -1 unit 7m 4s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_74.
          -1 unit 63m 38s hadoop-yarn-client in the patch failed with JDK v1.8.0_74.
          +1 unit 10m 8s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
          -1 unit 7m 4s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95.
          -1 unit 63m 50s hadoop-yarn-client in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          195m 54s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.client.TestGetGroups
          JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI
            org.apache.hadoop.yarn.client.api.impl.TestYarnClient
            org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
            org.apache.hadoop.yarn.client.api.impl.TestNMClient
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.client.TestGetGroups
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI
            org.apache.hadoop.yarn.client.api.impl.TestYarnClient
            org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
            org.apache.hadoop.yarn.client.api.impl.TestNMClient



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793341/YARN-4686.005.patch
          JIRA Issue YARN-4686
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3dd9ce79383b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5644137
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10777/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10777/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 7m 49s trunk passed +1 compile 2m 7s trunk passed with JDK v1.8.0_74 +1 compile 2m 28s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 39s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 44s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 0m 51s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 9s the patch passed +1 compile 2m 9s the patch passed with JDK v1.8.0_74 +1 javac 2m 9s the patch passed +1 compile 2m 23s the patch passed with JDK v1.7.0_95 +1 javac 2m 23s the patch passed -1 checkstyle 0m 35s hadoop-yarn-project/hadoop-yarn: patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30) +1 mvnsite 1m 12s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 33s the patch passed +1 javadoc 0m 53s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 54s the patch passed with JDK v1.7.0_95 +1 unit 9m 57s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74. -1 unit 7m 4s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_74. -1 unit 63m 38s hadoop-yarn-client in the patch failed with JDK v1.8.0_74. +1 unit 10m 8s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. -1 unit 7m 4s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95. -1 unit 63m 50s hadoop-yarn-client in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 195m 54s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.client.TestGetGroups JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI   org.apache.hadoop.yarn.client.api.impl.TestYarnClient   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient   org.apache.hadoop.yarn.client.api.impl.TestNMClient JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.client.TestGetGroups JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI   org.apache.hadoop.yarn.client.api.impl.TestYarnClient   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient   org.apache.hadoop.yarn.client.api.impl.TestNMClient Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793341/YARN-4686.005.patch JIRA Issue YARN-4686 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3dd9ce79383b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5644137 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10777/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10777/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/10777/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          As per my comment above:

          TestYarnCLI, TestAMRMClient, TestYarnClient, TestNMClient, and TestGetGroups are failing in multiple recent precommit builds YARN-4117, YARN-4630, YARN-4676.
          TestMiniYarnClusterNodeUtilization is tracked by YARN-4566.
          TestContainerManagerSecurity is failing on other recent precommit builds YARN-4117, YARN-4566.

          Those are the only tests that have failed and all all unrelated to the patch. Jason Lowe Karthik Kambatla Eric Payne please review the patch and give me your thoughts. Thanks!

          Show
          ebadger Eric Badger added a comment - As per my comment above: TestYarnCLI, TestAMRMClient, TestYarnClient, TestNMClient, and TestGetGroups are failing in multiple recent precommit builds YARN-4117 , YARN-4630 , YARN-4676 . TestMiniYarnClusterNodeUtilization is tracked by YARN-4566 . TestContainerManagerSecurity is failing on other recent precommit builds YARN-4117 , YARN-4566 . Those are the only tests that have failed and all all unrelated to the patch. Jason Lowe Karthik Kambatla Eric Payne please review the patch and give me your thoughts. Thanks!
          Hide
          eepayne Eric Payne added a comment -

          Eric Badger, thanks for updating this patch. Just one minor nit:
          In {{MiniYARNCluster#NodeManagerWrapper#serviceStart:

          +      if (nodeManagers[index].getServiceState() != STATE.STARTED) {
          +        // RM could have failed.
          +        throw new IOException("NodeManager " + index + " failed to start");
          

          The comment should be NM rather than RM

          Otherwise, looks good to me. Still interested in if Jason Lowe or Karthik Kambatla have comments, especially about removal of the (extra) threads in startResourceManager and serviceStart methods.

          Show
          eepayne Eric Payne added a comment - Eric Badger , thanks for updating this patch. Just one minor nit: In {{MiniYARNCluster#NodeManagerWrapper#serviceStart: + if (nodeManagers[index].getServiceState() != STATE.STARTED) { + // RM could have failed. + throw new IOException( "NodeManager " + index + " failed to start" ); The comment should be NM rather than RM Otherwise, looks good to me. Still interested in if Jason Lowe or Karthik Kambatla have comments, especially about removal of the (extra) threads in startResourceManager and serviceStart methods.
          Hide
          ebadger Eric Badger added a comment -

          Eric Payne good catch! I've switched that line in the comment in this updated patch.

          Show
          ebadger Eric Badger added a comment - Eric Payne good catch! I've switched that line in the comment in this updated patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 0m 56s Docker failed to build yetus/hadoop:0ca8df7.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793638/YARN-4686.006.patch
          JIRA Issue YARN-4686
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10794/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 56s Docker failed to build yetus/hadoop:0ca8df7. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793638/YARN-4686.006.patch JIRA Issue YARN-4686 Console output https://builds.apache.org/job/PreCommit-YARN-Build/10794/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 6m 39s trunk passed
          +1 compile 1m 49s trunk passed with JDK v1.8.0_74
          +1 compile 2m 6s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 10s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 1m 49s trunk passed
          +1 javadoc 0m 46s trunk passed with JDK v1.8.0_74
          +1 javadoc 0m 53s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 0m 58s the patch passed
          +1 compile 1m 50s the patch passed with JDK v1.8.0_74
          +1 javac 1m 50s the patch passed
          +1 compile 2m 5s the patch passed with JDK v1.7.0_95
          +1 javac 2m 5s the patch passed
          -1 checkstyle 0m 31s hadoop-yarn-project/hadoop-yarn: patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30)
          +1 mvnsite 1m 5s the patch passed
          +1 mvneclipse 0m 35s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 20s the patch passed
          +1 javadoc 0m 40s the patch passed with JDK v1.8.0_74
          +1 javadoc 0m 49s the patch passed with JDK v1.7.0_95
          +1 unit 9m 0s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74.
          -1 unit 6m 20s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_74.
          -1 unit 63m 18s hadoop-yarn-client in the patch failed with JDK v1.8.0_74.
          +1 unit 9m 33s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
          -1 unit 6m 30s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95.
          -1 unit 63m 32s hadoop-yarn-client in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          187m 38s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.client.TestGetGroups
          JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI
            org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
            org.apache.hadoop.yarn.client.api.impl.TestYarnClient
            org.apache.hadoop.yarn.client.api.impl.TestNMClient
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.client.TestGetGroups
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI
            org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
            org.apache.hadoop.yarn.client.api.impl.TestYarnClient
            org.apache.hadoop.yarn.client.api.impl.TestNMClient



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793638/YARN-4686.006.patch
          JIRA Issue YARN-4686
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 17950f10b756 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 605fdcb
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10797/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10797/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 6m 39s trunk passed +1 compile 1m 49s trunk passed with JDK v1.8.0_74 +1 compile 2m 6s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 10s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 1m 49s trunk passed +1 javadoc 0m 46s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 53s trunk passed with JDK v1.7.0_95 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 0m 58s the patch passed +1 compile 1m 50s the patch passed with JDK v1.8.0_74 +1 javac 1m 50s the patch passed +1 compile 2m 5s the patch passed with JDK v1.7.0_95 +1 javac 2m 5s the patch passed -1 checkstyle 0m 31s hadoop-yarn-project/hadoop-yarn: patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30) +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 20s the patch passed +1 javadoc 0m 40s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 49s the patch passed with JDK v1.7.0_95 +1 unit 9m 0s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74. -1 unit 6m 20s hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_74. -1 unit 63m 18s hadoop-yarn-client in the patch failed with JDK v1.8.0_74. +1 unit 9m 33s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. -1 unit 6m 30s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95. -1 unit 63m 32s hadoop-yarn-client in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 187m 38s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.client.TestGetGroups JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient   org.apache.hadoop.yarn.client.api.impl.TestYarnClient   org.apache.hadoop.yarn.client.api.impl.TestNMClient JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.client.TestGetGroups JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.yarn.client.cli.TestYarnCLI   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient   org.apache.hadoop.yarn.client.api.impl.TestYarnClient   org.apache.hadoop.yarn.client.api.impl.TestNMClient Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793638/YARN-4686.006.patch JIRA Issue YARN-4686 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 17950f10b756 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 605fdcb Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10797/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10797/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/10797/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          eepayne Eric Payne added a comment -

          Thanks, Eric Badger, for this patch.
          +1 LGTM.

          I will wait a day or so to give Jason Lowe, Karthik Kambatla, and others to comment. Then will commit if no further concerns.

          Show
          eepayne Eric Payne added a comment - Thanks, Eric Badger , for this patch. +1 LGTM. I will wait a day or so to give Jason Lowe , Karthik Kambatla , and others to comment. Then will commit if no further concerns.
          Hide
          kasha Karthik Kambatla added a comment -

          On a cursory look, the patch looks reasonable. One thing that caught my eye: are we explicitly transitioning the RM to active even when HA is not enabled? Is that required?

          Show
          kasha Karthik Kambatla added a comment - On a cursory look, the patch looks reasonable. One thing that caught my eye: are we explicitly transitioning the RM to active even when HA is not enabled? Is that required?
          Hide
          ebadger Eric Badger added a comment -

          Karthik Kambatla When HA is not enabled, the RM will be transitioned to active in the RM serviceStart method. So it is not necessary to transition it to active in the MiniYARNCluster serviceStart method. However, the transitionToActive method will not end up actually transitioning the RM again since it checks to make sure that it is not already active before proceeding. I believe that this makes the MiniYARNCluster start method a little bit cleaner since there are no extra checks. But, it may be more intuitive for the check to be in the MiniYARNCluster start so that it is obvious that the RM is only explicitly transitioned to active in HA setups. The check is made either way, it's just a matter of where we want the check to occur. Currently it is in the RM start method, but if you feel that it is better to put it in the MiniYARNCluster start method then we can add the check there.

          Show
          ebadger Eric Badger added a comment - Karthik Kambatla When HA is not enabled, the RM will be transitioned to active in the RM serviceStart method. So it is not necessary to transition it to active in the MiniYARNCluster serviceStart method. However, the transitionToActive method will not end up actually transitioning the RM again since it checks to make sure that it is not already active before proceeding. I believe that this makes the MiniYARNCluster start method a little bit cleaner since there are no extra checks. But, it may be more intuitive for the check to be in the MiniYARNCluster start so that it is obvious that the RM is only explicitly transitioned to active in HA setups. The check is made either way, it's just a matter of where we want the check to occur. Currently it is in the RM start method, but if you feel that it is better to put it in the MiniYARNCluster start method then we can add the check there.
          Hide
          jlowe Jason Lowe added a comment -

          Still interested in if Jason Lowe or Karthik Kambatla have comments, especially about removal of the (extra) threads in startResourceManager and serviceStart methods.

          The thread removal is key, IMHO. MiniYARNCluster was a source of flaky tests because those threads allowed the mini cluster to return from its start method before its subcomponents completed their start methods. That means tests that assumed the cluster was started after cluster.start() were making a bad assumption. Removing these threads means the cluster really is started after the start method, assuming the RM and NM start methods correctly return only after they have started.

          +1 patch looks good to me. I'm OK either way on the blind or checked transition to active since it's a fast no-op in the non-HA case. It will generate an extra "Already in active state" info message in the test logs but is otherwise benign.

          Show
          jlowe Jason Lowe added a comment - Still interested in if Jason Lowe or Karthik Kambatla have comments, especially about removal of the (extra) threads in startResourceManager and serviceStart methods. The thread removal is key, IMHO. MiniYARNCluster was a source of flaky tests because those threads allowed the mini cluster to return from its start method before its subcomponents completed their start methods. That means tests that assumed the cluster was started after cluster.start() were making a bad assumption. Removing these threads means the cluster really is started after the start method, assuming the RM and NM start methods correctly return only after they have started. +1 patch looks good to me. I'm OK either way on the blind or checked transition to active since it's a fast no-op in the non-HA case. It will generate an extra "Already in active state" info message in the test logs but is otherwise benign.
          Hide
          kasha Karthik Kambatla added a comment -

          Currently it is in the RM start method, but if you feel that it is better to put it in the MiniYARNCluster start method then we can add the check there.

          I was just wondering if we needed to transition to active. I am fine with the check being at either place. We can stick to this if it keeps the code clean.

          Show
          kasha Karthik Kambatla added a comment - Currently it is in the RM start method, but if you feel that it is better to put it in the MiniYARNCluster start method then we can add the check there. I was just wondering if we needed to transition to active. I am fine with the check being at either place. We can stick to this if it keeps the code clean.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9474 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9474/)
          YARN-4686. MiniYARNCluster.start() returns before cluster is completely (epayne: rev 92b7e0d41302b6b110927f99de5c2b4a4a93c5fd)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestMiniYARNClusterForHA.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9474 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9474/ ) YARN-4686 . MiniYARNCluster.start() returns before cluster is completely (epayne: rev 92b7e0d41302b6b110927f99de5c2b4a4a93c5fd) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestMiniYARNClusterForHA.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailover.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java
          Hide
          eepayne Eric Payne added a comment -

          Eric Badger, Thanks for the work you have done resolving this issue. I merged YARN-4686.006.patch to trunk and cherry-picked to branch-2 and branch-2.8. There were enough conflicts with the cherry-pick to branch-2.7 that I think it would be best if you provided a separate patch. The way it is written now, it has dependencies on JIRAs that were not backported to 2.7 (e.g., YARN-41).

          Show
          eepayne Eric Payne added a comment - Eric Badger , Thanks for the work you have done resolving this issue. I merged YARN-4686 .006.patch to trunk and cherry-picked to branch-2 and branch-2.8. There were enough conflicts with the cherry-pick to branch-2.7 that I think it would be best if you provided a separate patch. The way it is written now, it has dependencies on JIRAs that were not backported to 2.7 (e.g., YARN-41 ).
          Hide
          ebadger Eric Badger added a comment -

          Eric Payne Attaching the branch-2.7 patch. It passed all of the tests locally on my machine.

          Show
          ebadger Eric Badger added a comment - Eric Payne Attaching the branch-2.7 patch. It passed all of the tests locally on my machine.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 10s YARN-4686 does not apply to branch-2.7. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794247/YARN-4686-branch-2.7.006.patch
          JIRA Issue YARN-4686
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10820/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 10s YARN-4686 does not apply to branch-2.7. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794247/YARN-4686-branch-2.7.006.patch JIRA Issue YARN-4686 Console output https://builds.apache.org/job/PreCommit-YARN-Build/10820/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Attaching branch-2.7 patch after rebase. Must not have done that initially.

          Show
          ebadger Eric Badger added a comment - Attaching branch-2.7 patch after rebase. Must not have done that initially.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 7s YARN-4686 does not apply to branch-2.7. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794518/YARN-4686-branch-2.7.007.patch
          JIRA Issue YARN-4686
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10834/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s YARN-4686 does not apply to branch-2.7. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794518/YARN-4686-branch-2.7.007.patch JIRA Issue YARN-4686 Console output https://builds.apache.org/job/PreCommit-YARN-Build/10834/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          eepayne Eric Payne added a comment -

          Thanks, Eric Badger. I committed YARN-4686-branch-2.7.006.patch to branch-2.7.

          Show
          eepayne Eric Payne added a comment - Thanks, Eric Badger . I committed YARN-4686 -branch-2.7.006.patch to branch-2.7.
          Hide
          ebadger Eric Badger added a comment -

          Thanks, Eric Payne!

          Show
          ebadger Eric Badger added a comment - Thanks, Eric Payne !
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Eric Badger and Eric Payne, TestMRJobs#testJobWithChangePriority is failing after this issue. Would you fix the test failure?
          I've filed MAPREDUCE-6658 for fixing the failure.

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Eric Badger and Eric Payne , TestMRJobs#testJobWithChangePriority is failing after this issue. Would you fix the test failure? I've filed MAPREDUCE-6658 for fixing the failure.
          Hide
          eepayne Eric Payne added a comment -

          Hi Eric Badger and Eric Payne, TestMRJobs#testJobWithChangePriority is failing after this issue. Would you fix the test failure?
          I've filed MAPREDUCE-6658 for fixing the failure.

          Thanks, Akira Ajisaka for reporting this. Eric Badger is looking into this.

          Show
          eepayne Eric Payne added a comment - Hi Eric Badger and Eric Payne, TestMRJobs#testJobWithChangePriority is failing after this issue. Would you fix the test failure? I've filed MAPREDUCE-6658 for fixing the failure. Thanks, Akira Ajisaka for reporting this. Eric Badger is looking into this.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

            People

            • Assignee:
              ebadger Eric Badger
              Reporter:
              rohithsharma Rohith Sharma K S
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development