Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2966

TestNameNodeMetrics tests can fail under load

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: test
    • Labels:
      None
    • Environment:

      OS/X running intellij IDEA, firefox, winxp in a virtualbox.

      Description

      I've managed to recreate HDFS-540 and HDFS-2434 by the simple technique of running the HDFS tests on a desktop with out enough memory for all the programs trying to run. Things got swapped out and the tests failed as the DN heartbeats didn't come in on time.

      the tests both rely on waitForDeletion() to block the tests until the delete operation has completed, but all it does is sleep for the same number of seconds as there are datanodes. This is too brittle -it may work on a lightly-loaded system, but not on a system under heavy load where it is taking longer to replicate than expect.

      Immediate fix: double, triple, the sleep time?
      Better fix: have the thread block until all the DN heartbeats have finished.

      1. HDFS-2966.patch
        4 kB
        Steve Loughran
      2. HDFS-2966.patch
        4 kB
        Suresh Srinivas
      3. HDFS-2966.patch
        4 kB
        Steve Loughran
      4. HDFS-2966.patch
        4 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          Steve Loughran added a comment -

          stack trace against trunk

          -------------------------------------------------------------------------------
          Test set: org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          -------------------------------------------------------------------------------
          Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 53.526 sec <<< FAILURE!
          testCorruptBlock(org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics)  Time elapsed: 8.968 sec  <<< FAILURE!
          java.lang.AssertionError: Bad value for metric PendingReplicationBlocks expected:<0> but was:<1>
          	at org.junit.Assert.fail(Assert.java:91)
          	at org.junit.Assert.failNotEquals(Assert.java:645)
          	at org.junit.Assert.assertEquals(Assert.java:126)
          	at org.junit.Assert.assertEquals(Assert.java:470)
          	at org.apache.hadoop.test.MetricsAsserts.assertGauge(MetricsAsserts.java:185)
          	at org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics.testCorruptBlock(TestNameNodeMetrics.java:185)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
          	at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
          	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
          	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
          	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
          	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
          	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
          	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
          	at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
          	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
          	at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
          	at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
          
          Show
          Steve Loughran added a comment - stack trace against trunk ------------------------------------------------------------------------------- Test set: org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics ------------------------------------------------------------------------------- Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 53.526 sec <<< FAILURE! testCorruptBlock(org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics) Time elapsed: 8.968 sec <<< FAILURE! java.lang.AssertionError: Bad value for metric PendingReplicationBlocks expected:<0> but was:<1> at org.junit.Assert.fail(Assert.java:91) at org.junit.Assert.failNotEquals(Assert.java:645) at org.junit.Assert.assertEquals(Assert.java:126) at org.junit.Assert.assertEquals(Assert.java:470) at org.apache.hadoop.test.MetricsAsserts.assertGauge(MetricsAsserts.java:185) at org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics.testCorruptBlock(TestNameNodeMetrics.java:185) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
          Hide
          Steve Loughran added a comment -

          My planned solution to this is move from sleep-then-assert to sleep-poll-repeat for a longer period of time. If the state is reached sooner, the test finishes earlier, but if the machine is overloaded the test will stretch out. This may make it faster on some machines, as well as less brittle on others.

          Show
          Steve Loughran added a comment - My planned solution to this is move from sleep-then-assert to sleep-poll-repeat for a longer period of time. If the state is reached sooner, the test finishes earlier, but if the machine is overloaded the test will stretch out. This may make it faster on some machines, as well as less brittle on others.
          Hide
          Steve Loughran added a comment -

          A problem here is that the tests are not independent -the fs events from the previous test can still be trickling through the filesystem when the next test starts running.

          A simple poll/sleep cycle actually behaves worse, because it can exit too early; the state of the previous test is still there and the more recent changes aren't yet in the metrics.

          A sleep+ followup poll cycle would appear to be a better process, though it may still have problems under load that movind to per-test mini HDFS clusters would be required to fix.

          Show
          Steve Loughran added a comment - A problem here is that the tests are not independent -the fs events from the previous test can still be trickling through the filesystem when the next test starts running. A simple poll/sleep cycle actually behaves worse, because it can exit too early; the state of the previous test is still there and the more recent changes aren't yet in the metrics. A sleep+ followup poll cycle would appear to be a better process, though it may still have problems under load that movind to per-test mini HDFS clusters would be required to fix.
          Hide
          Steve Loughran added a comment -

          patch applies sleep (for the same delay as before) then poll+sleep for a limited set of retries before giving up.

          Provide the assertions are failing on the exit of the wait cycle, rather than on the initial state of the tests, this polling should significantly reduce the probability of failure under load.

          To reassure anyone worried that this polling would slow down the test run on a machine not under load, this does not appear to be the case.

          Before

          Running org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.042 sec
          

          On two runs after making changes, the elapsed times were 46.709 sec and 42.995 sec. This implies it takes about the same time.

          Show
          Steve Loughran added a comment - patch applies sleep (for the same delay as before) then poll+sleep for a limited set of retries before giving up. Provide the assertions are failing on the exit of the wait cycle, rather than on the initial state of the tests, this polling should significantly reduce the probability of failure under load. To reassure anyone worried that this polling would slow down the test run on a machine not under load, this does not appear to be the case. Before Running org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.042 sec On two runs after making changes, the elapsed times were 46.709 sec and 42.995 sec. This implies it takes about the same time.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12515090/HDFS-2966.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1885//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1885//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515090/HDFS-2966.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1885//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1885//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Hey Steve, patch looks pretty good. I agree this issue could stand to be improved. I've also seen spurious failures in this test.

          A few comments:

          1. In the spot where you call waitForGaugeValue for "FilesTotal", you also unnecessarily assert the value for FilesTotal.
          2. The name "waitForGaugeValue" seems a little misleading, since it's not a general-purpose method for gauges, but rather somewhat specific to gauges that are a function of DN metrics. Perhaps consider renaming it to something like "waitForDnMetricValue" ?
          3. Though the patch manages to get rid of the most race-prone sleeps (DN metrics), I don't think it will necessarily completely solve the issue for very slow VMs, since there are still several calls to updateMetrics. Can we completely remove the need for updateMetrics in this test, by waiting for a specific value as you've done here?
          Show
          Aaron T. Myers added a comment - Hey Steve, patch looks pretty good. I agree this issue could stand to be improved. I've also seen spurious failures in this test. A few comments: In the spot where you call waitForGaugeValue for "FilesTotal", you also unnecessarily assert the value for FilesTotal. The name "waitForGaugeValue" seems a little misleading, since it's not a general-purpose method for gauges, but rather somewhat specific to gauges that are a function of DN metrics . Perhaps consider renaming it to something like "waitForDnMetricValue" ? Though the patch manages to get rid of the most race-prone sleeps (DN metrics), I don't think it will necessarily completely solve the issue for very slow VMs, since there are still several calls to updateMetrics. Can we completely remove the need for updateMetrics in this test, by waiting for a specific value as you've done here?
          Hide
          Steve Loughran added a comment -

          point 1: probably trying to reduce change just to make sure I didn't accidentally remove an assertion. I will pull it.

          point 2: seems good.

          point 3: when I stripped it down too much the following tests fail -state propagates from one to the other. Moving to separate mini clusters could fix that but it would make things slower. That leaves "adding poll loops before each test case to ensure the FS is in the stable state before each test run. That's a harder thing to do and maybe something that can be put off unless this patch doesn't solve most people's problems.

          Show
          Steve Loughran added a comment - point 1: probably trying to reduce change just to make sure I didn't accidentally remove an assertion. I will pull it. point 2: seems good. point 3: when I stripped it down too much the following tests fail -state propagates from one to the other. Moving to separate mini clusters could fix that but it would make things slower. That leaves "adding poll loops before each test case to ensure the FS is in the stable state before each test run. That's a harder thing to do and maybe something that can be put off unless this patch doesn't solve most people's problems.
          Hide
          Aaron T. Myers added a comment -

          We can forego #3 for this JIRA, if you want. I may switch the whole test to use separate mini clusters in HDFS-2978 anyway, which would render this point moot.

          Show
          Aaron T. Myers added a comment - We can forego #3 for this JIRA, if you want. I may switch the whole test to use separate mini clusters in HDFS-2978 anyway, which would render this point moot.
          Hide
          Steve Loughran added a comment -

          updated patch. Rather than rename the method, I made the metric scope a parameter, so it can block for other metrics too.

          Show
          Steve Loughran added a comment - updated patch. Rather than rename the method, I made the metric scope a parameter, so it can block for other metrics too.
          Show
          Suresh Srinivas added a comment - Steve, please see HDFS-3002 - comment https://issues.apache.org/jira/browse/HDFS-3002?focusedCommentId=13214890&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13214890
          Hide
          Suresh Srinivas added a comment -

          Updated patch post HDFS-3002 changes.

          Show
          Suresh Srinivas added a comment - Updated patch post HDFS-3002 changes.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12516000/HDFS-2966.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1910//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1910//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516000/HDFS-2966.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1910//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1910//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Rather than rename the method, I made the metric scope a parameter, so it can block for other metrics too.

          I still find the name a tad misleading, since the method is still a little DN-specific. e.g. it retries based on the number of DNs, and sleeps a multiple of DFS_REPLICATION_INTERVAL. But, I don't feel super strongly about this point. Take it or leave it.

          +1, the patch looks good to me, assuming you don't want to address the above feedback.

          Show
          Aaron T. Myers added a comment - Rather than rename the method, I made the metric scope a parameter, so it can block for other metrics too. I still find the name a tad misleading, since the method is still a little DN-specific. e.g. it retries based on the number of DNs, and sleeps a multiple of DFS_REPLICATION_INTERVAL. But, I don't feel super strongly about this point. Take it or leave it. +1, the patch looks good to me, assuming you don't want to address the above feedback.
          Hide
          Steve Loughran added a comment -

          revised patch against trunk (some minor conflict; spaces?) and renamed method waitForDnMetricValue()

          Show
          Steve Loughran added a comment - revised patch against trunk (some minor conflict; spaces?) and renamed method waitForDnMetricValue()
          Hide
          Aaron T. Myers added a comment -

          +1, pending Jenkins.

          Show
          Aaron T. Myers added a comment - +1, pending Jenkins.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12517086/HDFS-2966.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1952//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1952//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517086/HDFS-2966.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1952//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1952//console This message is automatically generated.
          Hide
          Steve Loughran added a comment -

          fixed in trunk. Not patched 0.23.x as the test is out of sync with other changes, and it's not that important.

          Show
          Steve Loughran added a comment - fixed in trunk. Not patched 0.23.x as the test is out of sync with other changes, and it's not that important.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1931 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1931/)
          HDFS-2966 (Revision 1298820)

          Result = SUCCESS
          stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298820
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1931 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1931/ ) HDFS-2966 (Revision 1298820) Result = SUCCESS stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298820 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1856 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1856/)
          HDFS-2966 (Revision 1298820)

          Result = SUCCESS
          stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298820
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1856 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1856/ ) HDFS-2966 (Revision 1298820) Result = SUCCESS stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298820 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1865 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1865/)
          HDFS-2966 (Revision 1298820)

          Result = ABORTED
          stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298820
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1865 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1865/ ) HDFS-2966 (Revision 1298820) Result = ABORTED stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298820 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #980 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/980/)
          HDFS-2966 (Revision 1298820)

          Result = SUCCESS
          stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298820
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #980 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/980/ ) HDFS-2966 (Revision 1298820) Result = SUCCESS stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298820 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1015 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1015/)
          HDFS-2966 (Revision 1298820)

          Result = SUCCESS
          stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298820
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1015 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1015/ ) HDFS-2966 (Revision 1298820) Result = SUCCESS stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298820 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Hide
          Eli Collins added a comment -

          I saw this failure on branch-2, merged there.

          Show
          Eli Collins added a comment - I saw this failure on branch-2, merged there.
          Hide
          Trevor Robinson added a comment -

          I hit this on my last 2 builds of trunk. I don't see an open issue on it, so should I create a new issue or reopen this one (or HDFS-540)?

          testCorruptBlock(org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics)  Time elapsed: 7.082 sec  <<< FAILURE!
          java.lang.AssertionError: Bad value for metric PendingReplicationBlocks expected:<0> but was:<1>
                  at org.junit.Assert.fail(Assert.java:91)
                  at org.junit.Assert.failNotEquals(Assert.java:645)
                  at org.junit.Assert.assertEquals(Assert.java:126)
                  at org.junit.Assert.assertEquals(Assert.java:470)
                  at org.apache.hadoop.test.MetricsAsserts.assertGauge(MetricsAsserts.java:191)
                  at org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics.testCorruptBlock(TestNameNodeMetrics.java:186)
          
          Show
          Trevor Robinson added a comment - I hit this on my last 2 builds of trunk. I don't see an open issue on it, so should I create a new issue or reopen this one (or HDFS-540 )? testCorruptBlock(org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics) Time elapsed: 7.082 sec <<< FAILURE! java.lang.AssertionError: Bad value for metric PendingReplicationBlocks expected:<0> but was:<1> at org.junit.Assert.fail(Assert.java:91) at org.junit.Assert.failNotEquals(Assert.java:645) at org.junit.Assert.assertEquals(Assert.java:126) at org.junit.Assert.assertEquals(Assert.java:470) at org.apache.hadoop.test.MetricsAsserts.assertGauge(MetricsAsserts.java:191) at org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics.testCorruptBlock(TestNameNodeMetrics.java:186)
          Hide
          Aaron T. Myers added a comment -

          Hey Trevor, please open a new issue for it. Thanks a lot.

          Show
          Aaron T. Myers added a comment - Hey Trevor, please open a new issue for it. Thanks a lot.

            People

            • Assignee:
              Steve Loughran
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development