Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8911

NameNode Metric : Add Editlog counters as a JMX metric

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Today we log editlog metrics in the log. This JIRA proposes to expose those metrics via JMX.

        Issue Links

          Activity

          Hide
          anu Anu Engineer added a comment -

          Adds TotalSyncCount and TotalSyncTimes metrics to NameNodeMetrics

          Show
          anu Anu Engineer added a comment - Adds TotalSyncCount and TotalSyncTimes metrics to NameNodeMetrics
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 pending Jenkins.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 pending Jenkins.
          Hide
          andrew.wang Andrew Wang added a comment -

          This is a small correction, but the edit log is write behind not write ahead. Could you update the JIRA summary to just say "edit log" rather than WAL?

          Show
          andrew.wang Andrew Wang added a comment - This is a small correction, but the edit log is write behind not write ahead. Could you update the JIRA summary to just say "edit log" rather than WAL?
          Hide
          anu Anu Engineer added a comment -

          Andrew Wang Thanks for letting me know. I will fix that.

          Show
          anu Anu Engineer added a comment - Andrew Wang Thanks for letting me know. I will fix that.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 21m 38s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 57s There were no new javac warning messages.
          +1 javadoc 9m 57s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 site 3m 0s Site still builds.
          -1 checkstyle 2m 18s The applied patch generated 424 new checkstyle issues (total was 0, now 424).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 24s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 4m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 22m 40s Tests failed in hadoop-common.
          -1 hdfs tests 177m 34s Tests failed in hadoop-hdfs.
              251m 56s  



          Reason Tests
          Failed unit tests hadoop.net.TestNetUtils
            hadoop.ha.TestZKFailoverController
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          Timed out tests org.apache.hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12750932/HDFS-8911.001.patch
          Optional Tests site javadoc javac unit findbugs checkstyle
          git revision trunk / 71566e2
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12016/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12016/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12016/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12016/testReport/
          Java 1.7.0_55
          uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12016/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 21m 38s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 57s There were no new javac warning messages. +1 javadoc 9m 57s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 site 3m 0s Site still builds. -1 checkstyle 2m 18s The applied patch generated 424 new checkstyle issues (total was 0, now 424). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 24s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 4m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 22m 40s Tests failed in hadoop-common. -1 hdfs tests 177m 34s Tests failed in hadoop-hdfs.     251m 56s   Reason Tests Failed unit tests hadoop.net.TestNetUtils   hadoop.ha.TestZKFailoverController   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover Timed out tests org.apache.hadoop.cli.TestHDFSCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12750932/HDFS-8911.001.patch Optional Tests site javadoc javac unit findbugs checkstyle git revision trunk / 71566e2 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12016/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12016/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12016/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12016/testReport/ Java 1.7.0_55 uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12016/console This message was automatically generated.
          Hide
          anu Anu Engineer added a comment -

          Test failures does not seem to be related to this patch.

          Show
          anu Anu Engineer added a comment - Test failures does not seem to be related to this patch.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Committed to 2.8.0. Thanks Anu Engineer.

          Show
          arpitagarwal Arpit Agarwal added a comment - Committed to 2.8.0. Thanks Anu Engineer .
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8321 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8321/)
          HDFS-8911. NameNode Metric : Add Editlog counters as a JMX metric. (Contributed by Anu Engineer) (arp: rev 9c3571ea607f0953487464844ed0d46fdb3e9f90)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8321 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8321/ ) HDFS-8911 . NameNode Metric : Add Editlog counters as a JMX metric. (Contributed by Anu Engineer) (arp: rev 9c3571ea607f0953487464844ed0d46fdb3e9f90) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #293 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/293/)
          HDFS-8911. NameNode Metric : Add Editlog counters as a JMX metric. (Contributed by Anu Engineer) (arp: rev 9c3571ea607f0953487464844ed0d46fdb3e9f90)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
          • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #293 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/293/ ) HDFS-8911 . NameNode Metric : Add Editlog counters as a JMX metric. (Contributed by Anu Engineer) (arp: rev 9c3571ea607f0953487464844ed0d46fdb3e9f90) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1023 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1023/)
          HDFS-8911. NameNode Metric : Add Editlog counters as a JMX metric. (Contributed by Anu Engineer) (arp: rev 9c3571ea607f0953487464844ed0d46fdb3e9f90)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
          • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1023 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1023/ ) HDFS-8911 . NameNode Metric : Add Editlog counters as a JMX metric. (Contributed by Anu Engineer) (arp: rev 9c3571ea607f0953487464844ed0d46fdb3e9f90) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Hadoop-Hdfs-trunk #2220 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2220/)
          HDFS-8911. NameNode Metric : Add Editlog counters as a JMX metric. (Contributed by Anu Engineer) (arp: rev 9c3571ea607f0953487464844ed0d46fdb3e9f90)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
          • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk #2220 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2220/ ) HDFS-8911 . NameNode Metric : Add Editlog counters as a JMX metric. (Contributed by Anu Engineer) (arp: rev 9c3571ea607f0953487464844ed0d46fdb3e9f90) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #282 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/282/)
          HDFS-8911. NameNode Metric : Add Editlog counters as a JMX metric. (Contributed by Anu Engineer) (arp: rev 9c3571ea607f0953487464844ed0d46fdb3e9f90)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #282 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/282/ ) HDFS-8911 . NameNode Metric : Add Editlog counters as a JMX metric. (Contributed by Anu Engineer) (arp: rev 9c3571ea607f0953487464844ed0d46fdb3e9f90) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #290 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/290/)
          HDFS-8911. NameNode Metric : Add Editlog counters as a JMX metric. (Contributed by Anu Engineer) (arp: rev 9c3571ea607f0953487464844ed0d46fdb3e9f90)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java
          • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #290 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/290/ ) HDFS-8911 . NameNode Metric : Add Editlog counters as a JMX metric. (Contributed by Anu Engineer) (arp: rev 9c3571ea607f0953487464844ed0d46fdb3e9f90) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2239 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2239/)
          HDFS-8911. NameNode Metric : Add Editlog counters as a JMX metric. (Contributed by Anu Engineer) (arp: rev 9c3571ea607f0953487464844ed0d46fdb3e9f90)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java
          • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2239 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2239/ ) HDFS-8911 . NameNode Metric : Add Editlog counters as a JMX metric. (Contributed by Anu Engineer) (arp: rev 9c3571ea607f0953487464844ed0d46fdb3e9f90) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Review comment

          1. In java doc "+" should not come, I think this added by mistake

            /**
             +   * Return total number of syncs happened on this edit log.
             +   * @return long - count
             +   */
            public long getTotalSyncCount() {
          

          2. NPE in Namenode after this change

          I will fix both the issue in HDFS-8932

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Review comment 1. In java doc "+" should not come, I think this added by mistake /** + * Return total number of syncs happened on this edit log. + * @ return long - count + */ public long getTotalSyncCount() { 2. NPE in Namenode after this change I will fix both the issue in HDFS-8932

            People

            • Assignee:
              anu Anu Engineer
              Reporter:
              anu Anu Engineer
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development