Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5903

Fix race condition in TestResourceManagerAdministrationProtocolPBClientImpl beforeclass setup method

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: yarn
    • Labels:
      None

      Description

      This is essentially the same race condition as in YARN-5901, that is, resourcemanager.getServiceState() == STATE.STARTED does not guarantee resource manager is fully started.

      1. yarn5903.001.patch
        3 kB
        Haibo Chen
      2. YARN-5903.02.patch
        3 kB
        Haibo Chen
      3. YARN-5903.03.patch
        3 kB
        Haibo Chen

        Activity

        Hide
        haibochen Haibo Chen added a comment -

        Uploading the same fix as for YARN-5901. If this unreliable check is used often in the code base, we could extract it as a util method.

        Show
        haibochen Haibo Chen added a comment - Uploading the same fix as for YARN-5901 . If this unreliable check is used often in the code base, we could extract it as a util method.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 0s trunk passed
        +1 compile 0m 21s trunk passed
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 26s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 0m 37s trunk passed
        +1 javadoc 0m 15s trunk passed
        +1 mvninstall 0m 18s the patch passed
        +1 compile 0m 17s the patch passed
        +1 javac 0m 17s the patch passed
        +1 checkstyle 0m 11s the patch passed
        +1 mvnsite 0m 21s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 34s the patch passed
        +1 javadoc 0m 11s the patch passed
        +1 unit 16m 7s hadoop-yarn-client in the patch passed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        30m 16s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5903
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839423/yarn5903.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6afc0406f4e9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / bd37355
        Default Java 1.8.0_111
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13959/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13959/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 0s trunk passed +1 compile 0m 21s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 26s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 37s trunk passed +1 javadoc 0m 15s trunk passed +1 mvninstall 0m 18s the patch passed +1 compile 0m 17s the patch passed +1 javac 0m 17s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 34s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 16m 7s hadoop-yarn-client in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 30m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5903 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839423/yarn5903.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6afc0406f4e9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bd37355 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13959/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client Console output https://builds.apache.org/job/PreCommit-YARN-Build/13959/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        varun_saxena Varun Saxena added a comment -

        Do we get any test failure without this change ? IIUC client will retry if the service address is unreachable.
        You however are right about service state being set before RM actually starting.

        Show
        varun_saxena Varun Saxena added a comment - Do we get any test failure without this change ? IIUC client will retry if the service address is unreachable. You however are right about service state being set before RM actually starting.
        Hide
        haibochen Haibo Chen added a comment -

        Yes, we have seen consistent failures on some of our machines. My guess is the thread that starts the resource manager is always delayed on that platform. It is delayed so much so that client cannot reach the sever even with 10 retries.

        Show
        haibochen Haibo Chen added a comment - Yes, we have seen consistent failures on some of our machines. My guess is the thread that starts the resource manager is always delayed on that platform. It is delayed so much so that client cannot reach the sever even with 10 retries.
        Hide
        varun_saxena Varun Saxena added a comment -

        Yes, we have seen consistent failures on some of our machines. My guess is the thread that starts the resource manager is always delayed on that platform. It is delayed so much so that client cannot reach the sever even with 10 retries.

        Okay...The fix makes sense. Let me have a closer look.

        Show
        varun_saxena Varun Saxena added a comment - Yes, we have seen consistent failures on some of our machines. My guess is the thread that starts the resource manager is always delayed on that platform. It is delayed so much so that client cannot reach the sever even with 10 retries. Okay...The fix makes sense. Let me have a closer look.
        Hide
        varun_saxena Varun Saxena added a comment -

        Haibo Chen, previously we were throwing IOException if RM did not change to STARTED state.
        So shouldn't we check result of CountDownLatch#await and if its false, throw an IOException as that will indicate RM has not yet started.

        Show
        varun_saxena Varun Saxena added a comment - Haibo Chen , previously we were throwing IOException if RM did not change to STARTED state. So shouldn't we check result of CountDownLatch#await and if its false, throw an IOException as that will indicate RM has not yet started.
        Hide
        haibochen Haibo Chen added a comment -

        My bad. I incorrectly interpret the CountDownLatch API. WIll update the patch incorporate your comments.

        Show
        haibochen Haibo Chen added a comment - My bad. I incorrectly interpret the CountDownLatch API. WIll update the patch incorporate your comments.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 58s trunk passed
        +1 compile 0m 21s trunk passed
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 26s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 0m 30s trunk passed
        +1 javadoc 0m 16s trunk passed
        +1 mvninstall 0m 18s the patch passed
        +1 compile 0m 18s the patch passed
        +1 javac 0m 18s the patch passed
        +1 checkstyle 0m 12s the patch passed
        +1 mvnsite 0m 22s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 35s the patch passed
        +1 javadoc 0m 12s the patch passed
        +1 unit 16m 12s hadoop-yarn-client in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        29m 11s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5903
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840332/YARN-5903.02.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2f75562cecab 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0de0c32
        Default Java 1.8.0_111
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14067/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14067/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 58s trunk passed +1 compile 0m 21s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 26s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 16s trunk passed +1 mvninstall 0m 18s the patch passed +1 compile 0m 18s the patch passed +1 javac 0m 18s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 22s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 35s the patch passed +1 javadoc 0m 12s the patch passed +1 unit 16m 12s hadoop-yarn-client in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 29m 11s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5903 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840332/YARN-5903.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2f75562cecab 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0de0c32 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14067/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client Console output https://builds.apache.org/job/PreCommit-YARN-Build/14067/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        varun_saxena Varun Saxena added a comment -

        Probably we can make changes consistent with changes in YARN-5901.

        Show
        varun_saxena Varun Saxena added a comment - Probably we can make changes consistent with changes in YARN-5901 .
        Hide
        haibochen Haibo Chen added a comment -

        Upload another patch to make it consistent with YARN-5901 as Varun Saxena suggested.

        Show
        haibochen Haibo Chen added a comment - Upload another patch to make it consistent with YARN-5901 as Varun Saxena suggested.
        Hide
        varun_saxena Varun Saxena added a comment -

        +1 pending Jenkins.
        Will commit it later today unless there are further comments.

        Show
        varun_saxena Varun Saxena added a comment - +1 pending Jenkins. Will commit it later today unless there are further comments.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 15m 9s trunk passed
        +1 compile 0m 23s trunk passed
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 24s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 0m 32s trunk passed
        +1 javadoc 0m 16s trunk passed
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 19s the patch passed
        +1 javac 0m 19s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 24s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 44s the patch passed
        +1 javadoc 0m 13s the patch passed
        +1 unit 16m 51s hadoop-yarn-client in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        38m 37s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5903
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841788/YARN-5903.03.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7a9f90817049 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 736f54b
        Default Java 1.8.0_111
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14437/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14437/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 15m 9s trunk passed +1 compile 0m 23s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 24s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 32s trunk passed +1 javadoc 0m 16s trunk passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 19s the patch passed +1 javac 0m 19s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 44s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 16m 51s hadoop-yarn-client in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 38m 37s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5903 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841788/YARN-5903.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7a9f90817049 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 736f54b Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14437/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client Console output https://builds.apache.org/job/PreCommit-YARN-Build/14437/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11030 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11030/)
        YARN-5903. Fix race condition in (varunsaxena: rev 38e66d4d64f3c2e2bb43d8e5dca3866d672322b6)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceManagerAdministrationProtocolPBClientImpl.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11030 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11030/ ) YARN-5903 . Fix race condition in (varunsaxena: rev 38e66d4d64f3c2e2bb43d8e5dca3866d672322b6) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceManagerAdministrationProtocolPBClientImpl.java
        Hide
        varun_saxena Varun Saxena added a comment -

        Committed to trunk,branch-2.
        Thanks Haibo Chen for your contribution.

        Show
        varun_saxena Varun Saxena added a comment - Committed to trunk,branch-2. Thanks Haibo Chen for your contribution.

          People

          • Assignee:
            haibochen Haibo Chen
            Reporter:
            haibochen Haibo Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development