Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5492

TestSubmitApplicationWithRMHA is failing sporadically during precommit builds

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I've seen TestSubmitApplicationWithRMHA#testHandleRMHADuringSubmitApplicationCallWIthoutSavedApplicationState timeout on some recent YARN precommit builds.

        Activity

        Hide
        jlowe Jason Lowe added a comment -
        Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.69 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
        testHandleRMHADuringSubmitApplicationCallWithoutSavedApplicationState(org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA)  Time elapsed: 5.052 sec  <<< ERROR!
        java.lang.Exception: test timed out after 5000 milliseconds
        	at java.lang.Object.wait(Native Method)
        	at java.lang.Thread.join(Thread.java:1245)
        	at java.lang.Thread.join(Thread.java:1319)
        	at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.stopThreads(AbstractDelegationTokenSecretManager.java:627)
        	at org.apache.hadoop.yarn.server.resourcemanager.RMSecretManagerService.serviceStop(RMSecretManagerService.java:93)
        	at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
        	at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
        	at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
        	at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
        	at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
        	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStop(ResourceManager.java:728)
        	at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
        	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stopActiveServices(ResourceManager.java:1057)
        	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToStandby(ResourceManager.java:1112)
        	at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToStandby(AdminService.java:364)
        	at org.apache.hadoop.yarn.server.resourcemanager.RMHATestBase.explicitFailover(RMHATestBase.java:183)
        	at org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA.testHandleRMHADuringSubmitApplicationCallWithoutSavedApplicationState(TestSubmitApplicationWithRMHA.java:284)
        

        I'm guessing the 5 second timeout is too low.

        Show
        jlowe Jason Lowe added a comment - Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.69 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA testHandleRMHADuringSubmitApplicationCallWithoutSavedApplicationState(org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA) Time elapsed: 5.052 sec <<< ERROR! java.lang.Exception: test timed out after 5000 milliseconds at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1245) at java.lang.Thread.join(Thread.java:1319) at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.stopThreads(AbstractDelegationTokenSecretManager.java:627) at org.apache.hadoop.yarn.server.resourcemanager.RMSecretManagerService.serviceStop(RMSecretManagerService.java:93) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStop(ResourceManager.java:728) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stopActiveServices(ResourceManager.java:1057) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToStandby(ResourceManager.java:1112) at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToStandby(AdminService.java:364) at org.apache.hadoop.yarn.server.resourcemanager.RMHATestBase.explicitFailover(RMHATestBase.java:183) at org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA.testHandleRMHADuringSubmitApplicationCallWithoutSavedApplicationState(TestSubmitApplicationWithRMHA.java:284) I'm guessing the 5 second timeout is too low.
        Show
        vrushalic Vrushali C added a comment - Uploading patch 001. Increasing timeout to 50 seconds. The test code in branch-2.7 seems to have the timeout as 50 seconds. https://github.com/apache/hadoop/blob/branch-2.7/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSubmitApplicationWithRMHA.java#L263
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 39s trunk passed
        +1 compile 0m 31s trunk passed
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 0m 38s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 0m 56s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 30s the patch passed
        +1 compile 0m 29s the patch passed
        +1 javac 0m 29s the patch passed
        +1 checkstyle 0m 16s the patch passed
        +1 mvnsite 0m 35s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 1s the patch passed
        +1 javadoc 0m 18s the patch passed
        -1 unit 38m 23s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        52m 38s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822892/YARN-5492.001.patch
        JIRA Issue YARN-5492
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 72cd3e5b67a9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 85422bb
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/12703/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12703/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12703/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12703/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 39s trunk passed +1 compile 0m 31s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 56s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 29s the patch passed +1 javac 0m 29s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 1s the patch passed +1 javadoc 0m 18s the patch passed -1 unit 38m 23s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 52m 38s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822892/YARN-5492.001.patch JIRA Issue YARN-5492 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 72cd3e5b67a9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 85422bb Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/12703/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12703/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12703/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12703/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        It looks like test was running with too aggressive timeout. +1 LGTM

        Show
        rohithsharma Rohith Sharma K S added a comment - It looks like test was running with too aggressive timeout. +1 LGTM
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        The test code in branch-2.7 seems to have the timeout as 50 seconds.

        I am wondering how it got missed in trunk and branch-2!! Investigated and found that YARN-4312 patch committed only in branch-2.6 and branch-2.7.

        Show
        rohithsharma Rohith Sharma K S added a comment - The test code in branch-2.7 seems to have the timeout as 50 seconds. I am wondering how it got missed in trunk and branch-2!! Investigated and found that YARN-4312 patch committed only in branch-2.6 and branch-2.7.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        I have gone ahead with committing this patch in trunk/branch-2/branch-2.8 and branch-3.0.0-alpha1 to make changes are exist in latest branches.
        Thanks Vrushali C for the patch!!

        Show
        rohithsharma Rohith Sharma K S added a comment - I have gone ahead with committing this patch in trunk/branch-2/branch-2.8 and branch-3.0.0-alpha1 to make changes are exist in latest branches. Thanks Vrushali C for the patch!!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #10261 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10261/)
        YARN-5492. TestSubmitApplicationWithRMHA is failing sporadically during (rohithsharmaks: rev 5199db387d59f7233a0e52ac298df31e8ed8af20)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSubmitApplicationWithRMHA.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10261 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10261/ ) YARN-5492 . TestSubmitApplicationWithRMHA is failing sporadically during (rohithsharmaks: rev 5199db387d59f7233a0e52ac298df31e8ed8af20) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSubmitApplicationWithRMHA.java
        Hide
        vrushalic Vrushali C added a comment -
        Show
        vrushalic Vrushali C added a comment - Thanks Rohith Sharma K S !

          People

          • Assignee:
            vrushalic Vrushali C
            Reporter:
            jlowe Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development