Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 0.24.0
    • Fix Version/s: 2.0.0-alpha
    • Component/s: None
    • Labels:
      None
    1. hadoop-8204.txt
      14 kB
      Todd Lipcon
    2. hadoop-8204.txt
      12 kB
      Todd Lipcon

      Issue Links

        Activity

        Hide
        Todd Lipcon added a comment -

        Attached patch is on top of HADOOP-8193 and fixes the issue.

        The problem was that you can't concurrently call mocks and set up mock behavior in Mockito. So the health monitor would occasionally call the mock proxy at the same time as the test case was changing the behavior of the mock proxy.

        The fix is to pull some of the patch for HDFS-2185 into this one: HealthMonitor now monitors an HAServiceTarget, and the test implementation of HAServiceTarget has code to trigger different failure scenarios. The test now passes reliably for me.

        Show
        Todd Lipcon added a comment - Attached patch is on top of HADOOP-8193 and fixes the issue. The problem was that you can't concurrently call mocks and set up mock behavior in Mockito. So the health monitor would occasionally call the mock proxy at the same time as the test case was changing the behavior of the mock proxy. The fix is to pull some of the patch for HDFS-2185 into this one: HealthMonitor now monitors an HAServiceTarget, and the test implementation of HAServiceTarget has code to trigger different failure scenarios. The test now passes reliably for me.
        Hide
        Todd Lipcon added a comment -

        Rebased on trunk

        Show
        Todd Lipcon added a comment - Rebased on trunk
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12519829/hadoop-8204.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.fs.viewfs.TestViewFsTrash

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/773//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/773//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519829/hadoop-8204.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/773//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/773//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        +1, the patch looks good to me. I was able to get this test to fail about 1/5 times without this patch, and can't get it to fail with this patch.

        Show
        Aaron T. Myers added a comment - +1, the patch looks good to me. I was able to get this test to fail about 1/5 times without this patch, and can't get it to fail with this patch.
        Hide
        Todd Lipcon added a comment -

        Committed to trunk and 23, thanks for reviewing, Aaron.

        Show
        Todd Lipcon added a comment - Committed to trunk and 23, thanks for reviewing, Aaron.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #715 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/715/)
        HADOOP-8204. TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305198)

        Result = SUCCESS
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305198
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #715 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/715/ ) HADOOP-8204 . TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305198) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305198 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #725 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/725/)
        HADOOP-8204. TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305198)

        Result = SUCCESS
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305198
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #725 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/725/ ) HADOOP-8204 . TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305198) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305198 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2001 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2001/)
        HADOOP-8204. TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305199)

        Result = SUCCESS
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305199
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2001 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2001/ ) HADOOP-8204 . TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305199) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305199 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1927 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1927/)
        HADOOP-8204. TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305199)

        Result = SUCCESS
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305199
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1927 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1927/ ) HADOOP-8204 . TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305199) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305199 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1937 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1937/)
        HADOOP-8204. TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305199)

        Result = ABORTED
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305199
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1937 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1937/ ) HADOOP-8204 . TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305199) Result = ABORTED todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305199 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #734 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/734/)
        HADOOP-8204. TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305198)

        Result = ABORTED
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305198
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #734 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/734/ ) HADOOP-8204 . TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305198) Result = ABORTED todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305198 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #209 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/209/)
        HADOOP-8204. TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305198)

        Result = UNSTABLE
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305198
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #209 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/209/ ) HADOOP-8204 . TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305198) Result = UNSTABLE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305198 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #996 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/996/)
        HADOOP-8204. TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305199)

        Result = FAILURE
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305199
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #996 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/996/ ) HADOOP-8204 . TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305199) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305199 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #237 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/237/)
        HADOOP-8204. TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305198)

        Result = FAILURE
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305198
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #237 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/237/ ) HADOOP-8204 . TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305198) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305198 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1031 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1031/)
        HADOOP-8204. TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305199)

        Result = SUCCESS
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305199
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1031 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1031/ ) HADOOP-8204 . TestHealthMonitor fails occasionally. Contributed by Todd Lipcon. (Revision 1305199) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305199 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development