Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1318

HDFS Namenode and Datanode WebUI information needs to be accessible programmatically for scripts

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      resubmit the patch for HDFS1318 as Hudson was down last week.

      Description

      Currently Namenode and Datanode web page needs to be scraped by scripts to get this information. Having an interface where this structured information is provided, will help building scripts around it.

      1. HDFS-1318.y20.patch
        17 kB
        Tanping Wang
      2. HDFS-1318.y20.2.patch
        30 kB
        Tanping Wang
      3. HDFS1318.patch
        25 kB
        Tanping Wang
      4. HDFS1318.2.patch
        26 kB
        Tanping Wang

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #370 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/370/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #370 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/370/ )
          Hide
          Suresh Srinivas added a comment -

          Thank you Tanping for the patch.

          Show
          Suresh Srinivas added a comment - Thank you Tanping for the patch.
          Hide
          Suresh Srinivas added a comment -

          I committed the patch.

          Show
          Suresh Srinivas added a comment - I committed the patch.
          Hide
          Suresh Srinivas added a comment -

          Since hudson has some issues, ran the unit tests and testpatch. All the tests passed.

          Show
          Suresh Srinivas added a comment - Since hudson has some issues, ran the unit tests and testpatch. All the tests passed.
          Hide
          Tanping Wang added a comment -

          patch for HDFS1318:HDFS Namenode and Datanode WebUI information needs to be accessible programmatically for scripts

          Show
          Tanping Wang added a comment - patch for HDFS1318:HDFS Namenode and Datanode WebUI information needs to be accessible programmatically for scripts
          Hide
          Tanping Wang added a comment -

          add InterfaceAudience.Public and InterfaceStability.Evolving to DataNodeMxBean and NameNodeMXBean.

          Show
          Tanping Wang added a comment - add InterfaceAudience.Public and InterfaceStability.Evolving to DataNodeMxBean and NameNodeMXBean.
          Hide
          Suresh Srinivas added a comment -

          Tanping, please add InterfaceAudience.Public and InterfaceStability.Evolving to DataNodeMxBean and NameNodeMXBean.

          Show
          Suresh Srinivas added a comment - Tanping, please add InterfaceAudience.Public and InterfaceStability.Evolving to DataNodeMxBean and NameNodeMXBean.
          Hide
          Suresh Srinivas added a comment -

          Eli, the goal of HDFS-453 is to provide REST APIs for namenode WebUI. Adding REST interface is not the goal of this jira. Now that this jira exposes namenode and datanode web UI information, the JMX added in this jira could be used to provide REST interface or servlet based interface, more cleanly, without the need to access Namenode internal classes directly.

          If REST interface is no longer the goal, HDFS-453 can be closed.

          Show
          Suresh Srinivas added a comment - Eli, the goal of HDFS-453 is to provide REST APIs for namenode WebUI. Adding REST interface is not the goal of this jira. Now that this jira exposes namenode and datanode web UI information, the JMX added in this jira could be used to provide REST interface or servlet based interface, more cleanly, without the need to access Namenode internal classes directly. If REST interface is no longer the goal, HDFS-453 can be closed.
          Hide
          Tanping Wang added a comment -

          Description of the patch:
          Expose the following information about name node and data node through JMX to the end users.
          each datanode:
          hostname (string)
          rpcport (string)
          httpport (string)
          version (string): hadoop version
          each volume:

          • name: name of the volume
          • used: used dfs space
          • free: free dfs space
          • reserved: reserved space

          We group each volume information into a map object and serialize it to JSON string before sending to the client. This is an example of the volume information, {"/tmp/user/dfs/data/current/finalized":{"freeSpace":86118031360,"usedSpace":28672,"reservedSpace":0}}

          each namenode:
          hostname (string)
          version (string)
          used (long) : used dfs space
          free (long): free dfs space
          total (long): total dfs space
          safemode (string): safemode status
          isfinalize (boolean): is upgrade finalized.
          nondfsusedspace (long): total used space by data nodes fro non DFS purposes such as storing temporary files on the local file system
          percentused (float) : the total used space by data nodes as percentage of total capacity
          percentremaining (float): percentage of remaining space on the cluster
          totalblocks (long) : the total number of blocks on the cluster
          totalfiles (long): total number of files on the cluster
          threads (int): the number of threads

          each alive name node

          • name : host name of the alive name node
          • last contact : time since last contact in seconds
          • used: space used
            each dead name node:
          • name : host name of the dead name node
          • last contact: time since last contact in seconds
            each decommissioning node:
          • name: host name of the decommissioning node
          • under replicated blocks
          • decommission only replicas
          • under replicate in open files

          We group information of each alive/dead/decommissioning node into a map object and serialize it to JSON string before sending to the client. This is an example of alive node information, {"somehost.com":{"usedSpace":28672,"lastContact":1}}

          Two public interfaces are defined and implemented, i.e. NameNodeMXBean.java and DataNodeMXBean.java. The exposed name node, data node information can be used for monitoring purposes.

          Show
          Tanping Wang added a comment - Description of the patch: Expose the following information about name node and data node through JMX to the end users. each datanode: hostname (string) rpcport (string) httpport (string) version (string): hadoop version each volume: name: name of the volume used: used dfs space free: free dfs space reserved: reserved space We group each volume information into a map object and serialize it to JSON string before sending to the client. This is an example of the volume information, {"/tmp/user/dfs/data/current/finalized":{"freeSpace":86118031360,"usedSpace":28672,"reservedSpace":0}} each namenode: hostname (string) version (string) used (long) : used dfs space free (long): free dfs space total (long): total dfs space safemode (string): safemode status isfinalize (boolean): is upgrade finalized. nondfsusedspace (long): total used space by data nodes fro non DFS purposes such as storing temporary files on the local file system percentused (float) : the total used space by data nodes as percentage of total capacity percentremaining (float): percentage of remaining space on the cluster totalblocks (long) : the total number of blocks on the cluster totalfiles (long): total number of files on the cluster threads (int): the number of threads each alive name node name : host name of the alive name node last contact : time since last contact in seconds used: space used each dead name node: name : host name of the dead name node last contact: time since last contact in seconds each decommissioning node: name: host name of the decommissioning node under replicated blocks decommission only replicas under replicate in open files We group information of each alive/dead/decommissioning node into a map object and serialize it to JSON string before sending to the client. This is an example of alive node information, {"somehost.com":{"usedSpace":28672,"lastContact":1}} Two public interfaces are defined and implemented, i.e. NameNodeMXBean.java and DataNodeMXBean.java. The exposed name node, data node information can be used for monitoring purposes.
          Hide
          Tanping Wang added a comment -

          Thanks for pointing that out, Jakob. I have attached the new patch.

          Show
          Tanping Wang added a comment - Thanks for pointing that out, Jakob. I have attached the new patch.
          Hide
          Jakob Homan added a comment -

          The latest patch doesn't appear to have been synced with trunk and wipes out the changes introduced by HDFS-1150.

          Show
          Jakob Homan added a comment - The latest patch doesn't appear to have been synced with trunk and wipes out the changes introduced by HDFS-1150 .
          Hide
          Eli Collins added a comment -

          Would closing out HDFS-453 help address this?

          Show
          Eli Collins added a comment - Would closing out HDFS-453 help address this?
          Hide
          Tanping Wang added a comment -

          Generated a incremental patch based on Yahoo Y20 branch. Includes unit test cases.

          Show
          Tanping Wang added a comment - Generated a incremental patch based on Yahoo Y20 branch. Includes unit test cases.
          Hide
          Tanping Wang added a comment -

          add more statistics information for NN and DN.

          Show
          Tanping Wang added a comment - add more statistics information for NN and DN.

            People

            • Assignee:
              Tanping Wang
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development