Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3002

TestNameNodeMetrics need not wait for metrics update with new metrics framework

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 0.24.0
    • Fix Version/s: 3.0.0, 2.0.2-alpha
    • Component/s: test
    • Labels:
      None

      Description

      With older metrics framework, the namenode metrics was updated by replication thread. This required test having to wait for replication interval. This is no longer necessary with metrics2 framework.

      1. HDFS-3002.patch
        5 kB
        Suresh Srinivas

        Issue Links

          Activity

          Hide
          Aaron T. Myers added a comment -

          Hey Suresh, seems like this might be (at least partially) a duplicate of HDFS-2966?

          Show
          Aaron T. Myers added a comment - Hey Suresh, seems like this might be (at least partially) a duplicate of HDFS-2966 ?
          Hide
          Suresh Srinivas added a comment -

          In that jira, we are still adding wait time. I agree that our wait times for an even to happen could make the tests flaky. The main point of this jira is to not wait for updation of metrics. Perhaps I can add this comment to that jira and close this.

          Show
          Suresh Srinivas added a comment - In that jira, we are still adding wait time. I agree that our wait times for an even to happen could make the tests flaky. The main point of this jira is to not wait for updation of metrics. Perhaps I can add this comment to that jira and close this.
          Hide
          Suresh Srinivas added a comment -

          After looking at the patch on HDFS-2966, these are unrelated issues. I am removing updateMetrics() method calls.

          Show
          Suresh Srinivas added a comment - After looking at the patch on HDFS-2966 , these are unrelated issues. I am removing updateMetrics() method calls.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12515769/HDFS-3002.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1895//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1895//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515769/HDFS-3002.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1895//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1895//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          After looking at the patch on HDFS-2966, these are unrelated issues. I am removing updateMetrics() method calls.

          Not entirely unrelated. The patch for HDFS-2966 is also removing a few calls to updateMetrics(...), but only those which pertain to DN metrics, which still do take a little time to update. In your patch, you replaced those particular calls to direct calls to Thread.sleep(...).

          Regardless, the patch here looks good to me. +1. Once this is committed, we'll just have to rev the patch for HDFS-2966 a little bit, since I'm pretty sure they'll conflict slightly.

          Show
          Aaron T. Myers added a comment - After looking at the patch on HDFS-2966 , these are unrelated issues. I am removing updateMetrics() method calls. Not entirely unrelated. The patch for HDFS-2966 is also removing a few calls to updateMetrics(...), but only those which pertain to DN metrics, which still do take a little time to update. In your patch, you replaced those particular calls to direct calls to Thread.sleep(...). Regardless, the patch here looks good to me. +1. Once this is committed, we'll just have to rev the patch for HDFS-2966 a little bit, since I'm pretty sure they'll conflict slightly.
          Hide
          Suresh Srinivas added a comment -

          The Thread.sleep() part and waitForDeletion() is what HDFS-2966 is mainly concerned about. However, metrics is being used for detecting the completion of an activity it is waiting for completion. I can rebase HDFS-2966 after committing this patch.

          Show
          Suresh Srinivas added a comment - The Thread.sleep() part and waitForDeletion() is what HDFS-2966 is mainly concerned about. However, metrics is being used for detecting the completion of an activity it is waiting for completion. I can rebase HDFS-2966 after committing this patch.
          Hide
          Aaron T. Myers added a comment -

          Sounds good to me. Thanks a lot, Suresh.

          Show
          Aaron T. Myers added a comment - Sounds good to me. Thanks a lot, Suresh.
          Hide
          Steve Loughran added a comment -

          HDFS-2966 is independent of this as it is designed to handle the test failing on an overloaded system where the replication takes longer than the sleep times, so they both need to go in, though -2966 is a hard one to replicate. I will gladly let Suresh rebase that patch once this is in.

          Show
          Steve Loughran added a comment - HDFS-2966 is independent of this as it is designed to handle the test failing on an overloaded system where the replication takes longer than the sleep times, so they both need to go in, though -2966 is a hard one to replicate. I will gladly let Suresh rebase that patch once this is in.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1851 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1851/)
          HDFS-3002. TestNameNodeMetrics need not wait for metrics update. Contributed by Suresh Srinivas. (Revision 1293482)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293482
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1851 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1851/ ) HDFS-3002 . TestNameNodeMetrics need not wait for metrics update. Contributed by Suresh Srinivas. (Revision 1293482) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293482 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1777 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1777/)
          HDFS-3002. TestNameNodeMetrics need not wait for metrics update. Contributed by Suresh Srinivas. (Revision 1293482)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293482
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1777 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1777/ ) HDFS-3002 . TestNameNodeMetrics need not wait for metrics update. Contributed by Suresh Srinivas. (Revision 1293482) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293482 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1787 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1787/)
          HDFS-3002. TestNameNodeMetrics need not wait for metrics update. Contributed by Suresh Srinivas. (Revision 1293482)

          Result = ABORTED
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293482
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1787 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1787/ ) HDFS-3002 . TestNameNodeMetrics need not wait for metrics update. Contributed by Suresh Srinivas. (Revision 1293482) Result = ABORTED suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293482 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Hide
          Aaron T. Myers added a comment -

          Hey Suresh, I see that you committed this to trunk. I was about to resolve it for you, but then I noticed that you had the "affects version" set to 0.23.0, but with no "target version" set.

          Do you intend to commit this to 0.23 as well? Seems to me like we should.

          Show
          Aaron T. Myers added a comment - Hey Suresh, I see that you committed this to trunk. I was about to resolve it for you, but then I noticed that you had the "affects version" set to 0.23.0, but with no "target version" set. Do you intend to commit this to 0.23 as well? Seems to me like we should.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #966 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/966/)
          HDFS-3002. TestNameNodeMetrics need not wait for metrics update. Contributed by Suresh Srinivas. (Revision 1293482)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293482
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #966 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/966/ ) HDFS-3002 . TestNameNodeMetrics need not wait for metrics update. Contributed by Suresh Srinivas. (Revision 1293482) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293482 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1001 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1001/)
          HDFS-3002. TestNameNodeMetrics need not wait for metrics update. Contributed by Suresh Srinivas. (Revision 1293482)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293482
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1001 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1001/ ) HDFS-3002 . TestNameNodeMetrics need not wait for metrics update. Contributed by Suresh Srinivas. (Revision 1293482) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293482 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java

            People

            • Assignee:
              Suresh Srinivas
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development