Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1318

HDFS Namenode and Datanode WebUI information needs to be accessible programmatically for scripts

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      resubmit the patch for HDFS1318 as Hudson was down last week.

      Description

      Currently Namenode and Datanode web page needs to be scraped by scripts to get this information. Having an interface where this structured information is provided, will help building scripts around it.

      1. HDFS1318.2.patch
        26 kB
        Tanping Wang
      2. HDFS1318.patch
        25 kB
        Tanping Wang
      3. HDFS-1318.y20.2.patch
        30 kB
        Tanping Wang
      4. HDFS-1318.y20.patch
        17 kB
        Tanping Wang

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          5d 19h 5m 1 Tanping Wang 17/Aug/10 18:39
          Open Open Patch Available Patch Available
          19d 4h 38m 2 Tanping Wang 17/Aug/10 18:42
          Patch Available Patch Available Resolved Resolved
          5h 34m 1 Suresh Srinivas 18/Aug/10 00:16
          Resolved Resolved Closed Closed
          481d 7h 3m 1 Konstantin Shvachko 12/Dec/11 06:19
          Konstantin Shvachko made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Tanping Wang made changes -
          Link This issue is cloned as HDFS-1386 [ HDFS-1386 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #370 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/370/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #370 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/370/ )
          Hide
          Suresh Srinivas added a comment -

          Thank you Tanping for the patch.

          Show
          Suresh Srinivas added a comment - Thank you Tanping for the patch.
          Suresh Srinivas made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Suresh Srinivas added a comment -

          I committed the patch.

          Show
          Suresh Srinivas added a comment - I committed the patch.
          Hide
          Suresh Srinivas added a comment -

          Since hudson has some issues, ran the unit tests and testpatch. All the tests passed.

          Show
          Suresh Srinivas added a comment - Since hudson has some issues, ran the unit tests and testpatch. All the tests passed.
          Tanping Wang made changes -
          Release Note patch for HDFS1318:HDFS Namenode and Datanode WebUI information needs to be accessible programmatically for scripts (0.22.0) resubmit the patch for HDFS1318 as Hudson was down last week.
          Status Open [ 1 ] Patch Available [ 10002 ]
          Tanping Wang made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Assignee Tanping Wang [ tanping ]
          Tanping Wang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note patch for HDFS1318:HDFS Namenode and Datanode WebUI information needs to be accessible programmatically for scripts (0.22.0)
          Affects Version/s 0.22.0 [ 12314241 ]
          Hide
          Tanping Wang added a comment -

          patch for HDFS1318:HDFS Namenode and Datanode WebUI information needs to be accessible programmatically for scripts

          Show
          Tanping Wang added a comment - patch for HDFS1318:HDFS Namenode and Datanode WebUI information needs to be accessible programmatically for scripts
          Tanping Wang made changes -
          Attachment HDFS1318.2.patch [ 12451836 ]
          Hide
          Tanping Wang added a comment -

          add InterfaceAudience.Public and InterfaceStability.Evolving to DataNodeMxBean and NameNodeMXBean.

          Show
          Tanping Wang added a comment - add InterfaceAudience.Public and InterfaceStability.Evolving to DataNodeMxBean and NameNodeMXBean.
          Hide
          Suresh Srinivas added a comment -

          Tanping, please add InterfaceAudience.Public and InterfaceStability.Evolving to DataNodeMxBean and NameNodeMXBean.

          Show
          Suresh Srinivas added a comment - Tanping, please add InterfaceAudience.Public and InterfaceStability.Evolving to DataNodeMxBean and NameNodeMXBean.
          Hide
          Suresh Srinivas added a comment -

          Eli, the goal of HDFS-453 is to provide REST APIs for namenode WebUI. Adding REST interface is not the goal of this jira. Now that this jira exposes namenode and datanode web UI information, the JMX added in this jira could be used to provide REST interface or servlet based interface, more cleanly, without the need to access Namenode internal classes directly.

          If REST interface is no longer the goal, HDFS-453 can be closed.

          Show
          Suresh Srinivas added a comment - Eli, the goal of HDFS-453 is to provide REST APIs for namenode WebUI. Adding REST interface is not the goal of this jira. Now that this jira exposes namenode and datanode web UI information, the JMX added in this jira could be used to provide REST interface or servlet based interface, more cleanly, without the need to access Namenode internal classes directly. If REST interface is no longer the goal, HDFS-453 can be closed.
          Hide
          Tanping Wang added a comment -

          Description of the patch:
          Expose the following information about name node and data node through JMX to the end users.
          each datanode:
          hostname (string)
          rpcport (string)
          httpport (string)
          version (string): hadoop version
          each volume:

          • name: name of the volume
          • used: used dfs space
          • free: free dfs space
          • reserved: reserved space

          We group each volume information into a map object and serialize it to JSON string before sending to the client. This is an example of the volume information, {"/tmp/user/dfs/data/current/finalized":{"freeSpace":86118031360,"usedSpace":28672,"reservedSpace":0}}

          each namenode:
          hostname (string)
          version (string)
          used (long) : used dfs space
          free (long): free dfs space
          total (long): total dfs space
          safemode (string): safemode status
          isfinalize (boolean): is upgrade finalized.
          nondfsusedspace (long): total used space by data nodes fro non DFS purposes such as storing temporary files on the local file system
          percentused (float) : the total used space by data nodes as percentage of total capacity
          percentremaining (float): percentage of remaining space on the cluster
          totalblocks (long) : the total number of blocks on the cluster
          totalfiles (long): total number of files on the cluster
          threads (int): the number of threads

          each alive name node

          • name : host name of the alive name node
          • last contact : time since last contact in seconds
          • used: space used
            each dead name node:
          • name : host name of the dead name node
          • last contact: time since last contact in seconds
            each decommissioning node:
          • name: host name of the decommissioning node
          • under replicated blocks
          • decommission only replicas
          • under replicate in open files

          We group information of each alive/dead/decommissioning node into a map object and serialize it to JSON string before sending to the client. This is an example of alive node information, {"somehost.com":{"usedSpace":28672,"lastContact":1}}

          Two public interfaces are defined and implemented, i.e. NameNodeMXBean.java and DataNodeMXBean.java. The exposed name node, data node information can be used for monitoring purposes.

          Show
          Tanping Wang added a comment - Description of the patch: Expose the following information about name node and data node through JMX to the end users. each datanode: hostname (string) rpcport (string) httpport (string) version (string): hadoop version each volume: name: name of the volume used: used dfs space free: free dfs space reserved: reserved space We group each volume information into a map object and serialize it to JSON string before sending to the client. This is an example of the volume information, {"/tmp/user/dfs/data/current/finalized":{"freeSpace":86118031360,"usedSpace":28672,"reservedSpace":0}} each namenode: hostname (string) version (string) used (long) : used dfs space free (long): free dfs space total (long): total dfs space safemode (string): safemode status isfinalize (boolean): is upgrade finalized. nondfsusedspace (long): total used space by data nodes fro non DFS purposes such as storing temporary files on the local file system percentused (float) : the total used space by data nodes as percentage of total capacity percentremaining (float): percentage of remaining space on the cluster totalblocks (long) : the total number of blocks on the cluster totalfiles (long): total number of files on the cluster threads (int): the number of threads each alive name node name : host name of the alive name node last contact : time since last contact in seconds used: space used each dead name node: name : host name of the dead name node last contact: time since last contact in seconds each decommissioning node: name: host name of the decommissioning node under replicated blocks decommission only replicas under replicate in open files We group information of each alive/dead/decommissioning node into a map object and serialize it to JSON string before sending to the client. This is an example of alive node information, {"somehost.com":{"usedSpace":28672,"lastContact":1}} Two public interfaces are defined and implemented, i.e. NameNodeMXBean.java and DataNodeMXBean.java. The exposed name node, data node information can be used for monitoring purposes.
          Tanping Wang made changes -
          Attachment HDFS1318.patch [ 12451263 ]
          Hide
          Tanping Wang added a comment -

          Thanks for pointing that out, Jakob. I have attached the new patch.

          Show
          Tanping Wang added a comment - Thanks for pointing that out, Jakob. I have attached the new patch.
          Tanping Wang made changes -
          Attachment HDFS1318.patch [ 12451240 ]
          Hide
          Jakob Homan added a comment -

          The latest patch doesn't appear to have been synced with trunk and wipes out the changes introduced by HDFS-1150.

          Show
          Jakob Homan added a comment - The latest patch doesn't appear to have been synced with trunk and wipes out the changes introduced by HDFS-1150 .
          Tanping Wang made changes -
          Attachment HDFS1318.patch [ 12451240 ]
          Tanping Wang made changes -
          Attachment HDFS-1318.y20.2.patch [ 12451080 ]
          Tanping Wang made changes -
          Attachment HDFS-1318.y20.incremental.patch [ 12451053 ]
          Hide
          Eli Collins added a comment -

          Would closing out HDFS-453 help address this?

          Show
          Eli Collins added a comment - Would closing out HDFS-453 help address this?
          Tanping Wang made changes -
          Attachment HDFS-1318.y20.incremental.patch [ 12451053 ]
          Hide
          Tanping Wang added a comment -

          Generated a incremental patch based on Yahoo Y20 branch. Includes unit test cases.

          Show
          Tanping Wang added a comment - Generated a incremental patch based on Yahoo Y20 branch. Includes unit test cases.
          Tanping Wang made changes -
          Attachment metricsJira.patch [ 12450340 ]
          Tanping Wang made changes -
          Attachment HDFS-1318.y20.patch [ 12450341 ]
          Tanping Wang made changes -
          Attachment metricsJira.patch [ 12450340 ]
          Hide
          Tanping Wang added a comment -

          add more statistics information for NN and DN.

          Show
          Tanping Wang added a comment - add more statistics information for NN and DN.
          Suresh Srinivas made changes -
          Field Original Value New Value
          Fix Version/s 0.22.0 [ 12314241 ]
          Suresh Srinivas created issue -

            People

            • Assignee:
              Tanping Wang
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development