Hadoop YARN
  1. Hadoop YARN
  2. YARN-206

TestApplicationCleanup.testContainerCleanup occasionally fails

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.3-alpha, 0.23.5
    • Component/s: resourcemanager
    • Labels:
      None

      Description

      testContainerCleanup is occasionally failing with the error:

      testContainerCleanup(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup): expected:<2> but was:<1>

      1. YARN-206.patch
        7 kB
        Jason Lowe
      2. YARN-206-branch23.patch
        6 kB
        Jason Lowe

        Activity

        Hide
        Jason Lowe added a comment -

        Stacktrace from the failure:

        testContainerCleanup(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup)  Time elapsed: 3610 sec  <<< FAILURE!
        junit.framework.AssertionFailedError: expected:<2> but was:<1>
        	at junit.framework.Assert.fail(Assert.java:47)
        	at junit.framework.Assert.failNotEquals(Assert.java:283)
        	at junit.framework.Assert.assertEquals(Assert.java:64)
        	at junit.framework.Assert.assertEquals(Assert.java:195)
        	at junit.framework.Assert.assertEquals(Assert.java:201)
        	at org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup.testContainerCleanup(TestApplicationCleanup.java:182)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
        	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
        	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
        	at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
        	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
        	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
        	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
        	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
        	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
        	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
        	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
        	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
        	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
        	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
        	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
        	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
        

        The problem is that the assert is checking for conts.size() when it needs to check for contReceived which is the sum of all containers received so far. Test only passes if we receive the expected number of containers in one request – if it takes more than one request then the test fails.

        Show
        Jason Lowe added a comment - Stacktrace from the failure: testContainerCleanup(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup) Time elapsed: 3610 sec <<< FAILURE! junit.framework.AssertionFailedError: expected:<2> but was:<1> at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.failNotEquals(Assert.java:283) at junit.framework.Assert.assertEquals(Assert.java:64) at junit.framework.Assert.assertEquals(Assert.java:195) at junit.framework.Assert.assertEquals(Assert.java:201) at org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup.testContainerCleanup(TestApplicationCleanup.java:182) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) The problem is that the assert is checking for conts.size() when it needs to check for contReceived which is the sum of all containers received so far. Test only passes if we receive the expected number of containers in one request – if it takes more than one request then the test fails.
        Hide
        Jason Lowe added a comment -

        Patch to assert on the proper values. Also fixed where and how long the tests were sleeping since the tests were often sleeping unnecessarily. This shaved a few seconds off the total test runtime for a typical test run.

        Show
        Jason Lowe added a comment - Patch to assert on the proper values. Also fixed where and how long the tests were sleeping since the tests were often sleeping unnecessarily. This shaved a few seconds off the total test runtime for a typical test run.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12552559/YARN-206.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/143//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/143//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12552559/YARN-206.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/143//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/143//console This message is automatically generated.
        Hide
        Jonathan Eagles added a comment -

        +1 LGTM. I have verified that this test does run consistently now as compared to before. Checking this in.

        Show
        Jonathan Eagles added a comment - +1 LGTM. I have verified that this test does run consistently now as compared to before. Checking this in.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #2993 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2993/)
        YARN-206. TestApplicationCleanup.testContainerCleanup occasionally fails. (jlowe via jeagles) (Revision 1407606)

        Result = SUCCESS
        jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407606
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #2993 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2993/ ) YARN-206 . TestApplicationCleanup.testContainerCleanup occasionally fails. (jlowe via jeagles) (Revision 1407606) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407606 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
        Hide
        Robert Joseph Evans added a comment -

        This appears to have broken branch-0.23

        Show
        Robert Joseph Evans added a comment - This appears to have broken branch-0.23
        Hide
        Robert Joseph Evans added a comment -

        I am pulling this out.

        Show
        Robert Joseph Evans added a comment - I am pulling this out.
        Hide
        Jason Lowe added a comment -

        Sorry, didn't realize there was a need for a branch-0.23 patch. Here's the patch for 0.23.

        Show
        Jason Lowe added a comment - Sorry, didn't realize there was a need for a branch-0.23 patch. Here's the patch for 0.23.
        Hide
        Robert Joseph Evans added a comment -

        Thanks for the new Patch Jason,

        It looks good. Looks like an extra heartbeat was removed and the compilation issue was fixed so that the variable has a type now. It looks good and the test passes.

        +1

        I'll check it in.

        Show
        Robert Joseph Evans added a comment - Thanks for the new Patch Jason, It looks good. Looks like an extra heartbeat was removed and the compilation issue was fixed so that the variable has a type now. It looks good and the test passes. +1 I'll check it in.
        Hide
        Robert Joseph Evans added a comment -

        Fix for branch-0.23 has been checked in.

        Show
        Robert Joseph Evans added a comment - Fix for branch-0.23 has been checked in.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #32 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/32/)
        YARN-206. TestApplicationCleanup.testContainerCleanup occasionally fails. (jlowe via jeagles) (Revision 1407606)

        Result = SUCCESS
        jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407606
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #32 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/32/ ) YARN-206 . TestApplicationCleanup.testContainerCleanup occasionally fails. (jlowe via jeagles) (Revision 1407606) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407606 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #431 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/431/)
        YARN-206. TestApplicationCleanup.testContainerCleanup occasionally fails (jlowe via bobby) (Revision 1407697)
        svn merge -r 1407610:1407609 . reverted YARN-206 (Revision 1407688)
        YARN-206. TestApplicationCleanup.testContainerCleanup occasionally fails. (jlowe via jeagles) (Revision 1407610)

        Result = UNSTABLE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407697
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java

        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407688
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java

        jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407610
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #431 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/431/ ) YARN-206 . TestApplicationCleanup.testContainerCleanup occasionally fails (jlowe via bobby) (Revision 1407697) svn merge -r 1407610:1407609 . reverted YARN-206 (Revision 1407688) YARN-206 . TestApplicationCleanup.testContainerCleanup occasionally fails. (jlowe via jeagles) (Revision 1407610) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407697 Files : /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407688 Files : /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407610 Files : /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1222 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1222/)
        YARN-206. TestApplicationCleanup.testContainerCleanup occasionally fails. (jlowe via jeagles) (Revision 1407606)

        Result = SUCCESS
        jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407606
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1222 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1222/ ) YARN-206 . TestApplicationCleanup.testContainerCleanup occasionally fails. (jlowe via jeagles) (Revision 1407606) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407606 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1253 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1253/)
        YARN-206. TestApplicationCleanup.testContainerCleanup occasionally fails. (jlowe via jeagles) (Revision 1407606)

        Result = FAILURE
        jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407606
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1253 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1253/ ) YARN-206 . TestApplicationCleanup.testContainerCleanup occasionally fails. (jlowe via jeagles) (Revision 1407606) Result = FAILURE jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407606 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java

          People

          • Assignee:
            Jason Lowe
            Reporter:
            Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development