Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2083

Adopt JMXJsonServlet into HDFS in order to query statistics

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: None
    • Labels:
      None

      Description

      HADOOP-7144 added JMXJsonServlet into Common. It gives the capability to query statistics and metrics exposed via JMX to be queried through HTTP. We adopt this into HDFS. This provides the alternative solution to HDFS-1874.

      1. HDFS-2083.4.patch
        14 kB
        Tanping Wang
      2. HDFS-2083.3.patch
        13 kB
        Tanping Wang
      3. HDFS-2083.2.patch
        13 kB
        Tanping Wang
      4. HDFS-2083.patch
        9 kB
        Tanping Wang

        Issue Links

          Activity

          Hide
          Suresh Srinivas added a comment -
          1. Minor: Make this a staic string? "/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo"
          2. Why do you need to create new string in readOutput() out.append(new String(buffer, 0, len));
          3. Every time you need a property, you are querying mbean. Can you do it only once and hold on to the response?
          4. NamenodeMXBeanHelper javadoc needs to be updated (It still talks about JMX, also look at other references that talks about JMX access)
          5. Stream from URLConnection needs to be closed.
          Show
          Suresh Srinivas added a comment - Minor: Make this a staic string? "/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo" Why do you need to create new string in readOutput() out.append(new String(buffer, 0, len)); Every time you need a property, you are querying mbean. Can you do it only once and hold on to the response? NamenodeMXBeanHelper javadoc needs to be updated (It still talks about JMX, also look at other references that talks about JMX access) Stream from URLConnection needs to be closed.
          Hide
          Tanping Wang added a comment -

          upload a new patch to address the review comments.

          Show
          Tanping Wang added a comment - upload a new patch to address the review comments.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12483327/HDFS-2083.2.patch
          against trunk revision 1138098.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/809//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/809//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/809//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483327/HDFS-2083.2.patch against trunk revision 1138098. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/809//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/809//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/809//console This message is automatically generated.
          Hide
          Tanping Wang added a comment -

          Upload a new patch to fix the find bug warning.

          Show
          Tanping Wang added a comment - Upload a new patch to fix the find bug warning.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12483345/HDFS-2083.3.patch
          against trunk revision 1138098.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/810//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/810//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/810//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483345/HDFS-2083.3.patch against trunk revision 1138098. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/810//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/810//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/810//console This message is automatically generated.
          Hide
          Tanping Wang added a comment -

          No unit test included because this is web UI. Tested it manually.

          Show
          Tanping Wang added a comment - No unit test included because this is web UI. Tested it manually.
          Hide
          Suresh Srinivas added a comment -

          Minor comments:

          1. "Namenode over to get attributed" - over what? It would be good add some description that this connects over http to get Namenode JMX attributes.
          2. Decommissionnode -> DecommissionNode (at many places)
          Show
          Suresh Srinivas added a comment - Minor comments: "Namenode over to get attributed" - over what? It would be good add some description that this connects over http to get Namenode JMX attributes. Decommissionnode -> DecommissionNode (at many places)
          Hide
          Suresh Srinivas added a comment -

          +1 for the patch

          Show
          Suresh Srinivas added a comment - +1 for the patch
          Hide
          Tanping Wang added a comment -

          Upload a new patch to address the review comments.

          Show
          Tanping Wang added a comment - Upload a new patch to address the review comments.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12483464/HDFS-2083.4.patch
          against trunk revision 1138262.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/817//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/817//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/817//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483464/HDFS-2083.4.patch against trunk revision 1138262. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/817//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/817//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/817//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #754 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/754/)
          HDFS-2083 Query JMX statistics over http via JMXJsonServlet. (tanping)

          tanping : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1138645
          Files :

          • /hadoop/common/trunk/hdfs/CHANGES.txt
          • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/ClusterJspHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #754 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/754/ ) HDFS-2083 Query JMX statistics over http via JMXJsonServlet. (tanping) tanping : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1138645 Files : /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/ClusterJspHelper.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #704 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/704/)
          HDFS-2083 Query JMX statistics over http via JMXJsonServlet. (tanping)

          tanping : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1138645
          Files :

          • /hadoop/common/trunk/hdfs/CHANGES.txt
          • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/ClusterJspHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #704 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/704/ ) HDFS-2083 Query JMX statistics over http via JMXJsonServlet. (tanping) tanping : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1138645 Files : /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/ClusterJspHelper.java

            People

            • Assignee:
              Tanping Wang
              Reporter:
              Tanping Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development