Hadoop Common
  1. Hadoop Common
  2. HADOOP-5094

Show dead nodes information in dfsadmin -report

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.18.2
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Changed df dfsadmin -report to list live and dead nodes, and attempt to resolve the hostname of datanode ip addresses.

      Description

      As part of operations responsibility to bring back dead nodes, it will be good to have a quick way to obtain a list of dead data nodes.
      The current way is to scrape the namenode web UI page and parse that information, but this creates load on the namenode.
      In search of a less costly way, I noticed dfsadmin -report only reports data nodes with State: "In Service" and "Decommission in progress" get listed.
      Asking for a cheap way to obtain a list of dead nodes.

      In addition, can the following requests be reviewed for additional enhancement and changes to dfsadmin -report.

      • Consistent formatting output in "Remaining raw bytes:" for the data nodes should have a space between the exact value and the parenthesized value.
        Sample:
        Total raw bytes: 3842232975360 (3.49 TB)
        Remaining raw bytes: 146090593065(136.06 GB)
        Used raw bytes: 3240864964620 (2.95 TB)
      • Include the running version of Hadoop.
      • What is the meaning of "Total effective bytes"?
      • Display the hostname instead of the IP address for the data node (toggle option?)
      1. DfsAdminDeadNode_testCases.html
        3 kB
        gary murry
      2. DfsAdminDeadNode_testCases.html
        2 kB
        gary murry
      3. HADOOP-5094.patch
        6 kB
        Jakob Homan
      4. HADOOP-5094.patch
        6 kB
        Jakob Homan
      5. HADOOP-5094.patch
        4 kB
        Jakob Homan

        Issue Links

          Activity

          Allen Wittenauer made changes -
          Link This issue duplicates HDFS-363 [ HDFS-363 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          gary murry made changes -
          Attachment DfsAdminDeadNode_testCases.html [ 12424082 ]
          Robert Chansler made changes -
          Release Note Update the output of dfsadmin -report to delineate the live and dead nodes, as well as attempt to resolve the hostname of datanode ip addresses. Minor formatting changes. Changed df dfsadmin -report to list live and dead nodes, and attempt to resolve the hostname of datanode ip addresses.
          gary murry made changes -
          Attachment DfsAdminDeadNode_testCases.html [ 12420710 ]
          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]
          Jakob Homan made changes -
          Release Note Update the output of dfsadmin -report to delineate the live and dead nodes, as well as attempt to resolve the hostname of datanode ip addresses. Minor formatting changes.
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Tsz Wo Nicholas Sze made changes -
          Resolution Fixed [ 1 ]
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Tsz Wo Nicholas Sze made changes -
          Issue Type New Feature [ 2 ] Improvement [ 4 ]
          Hadoop Flags [Incompatible change] [Incompatible change, Reviewed]
          Jakob Homan made changes -
          Attachment HADOOP-5094.patch [ 12399407 ]
          Jakob Homan made changes -
          Link This issue incorporates HADOOP-2937 [ HADOOP-2937 ]
          Jakob Homan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jakob Homan made changes -
          Attachment HADOOP-5094.patch [ 12399197 ]
          Jakob Homan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jakob Homan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jakob Homan made changes -
          Hadoop Flags [Incompatible change]
          Description
          As part of operations responsibility to bring back dead nodes, it will be good to have a quick way to obtain a list of dead data nodes.
          The current way is to scrape the namenode web UI page and parse that information, but this creates load on the namenode.
          In search of a less costly way, I noticed dfsadmin -report only reports data nodes with State: "In Service" and "Decommission in progress" get listed.
          Asking for a cheap way to obtain a list of dead nodes.

          In addition, can the following requests be reviewed for additional enhancement and changes to dfsadmin -report.

          - Consistent formatting output in "Remaining raw bytes:" for the data nodes should have a space between the exact value and the parenthesized value.
          Sample:
          Total raw bytes: 3842232975360 (3.49 TB)
          Remaining raw bytes: 146090593065(136.06 GB)
          Used raw bytes: 3240864964620 (2.95 TB)

          - Include the running version of Hadoop.

          - What is the meaning of "Total effective bytes"?

          - Display the hostname instead of the IP address for the data node (toggle option?)
          As part of operations responsibility to bring back dead nodes, it will be good to have a quick way to obtain a list of dead data nodes.
          The current way is to scrape the namenode web UI page and parse that information, but this creates load on the namenode.
          In search of a less costly way, I noticed dfsadmin -report only reports data nodes with State: "In Service" and "Decommission in progress" get listed.
          Asking for a cheap way to obtain a list of dead nodes.

          In addition, can the following requests be reviewed for additional enhancement and changes to dfsadmin -report.

          - Consistent formatting output in "Remaining raw bytes:" for the data nodes should have a space between the exact value and the parenthesized value.
          Sample:
          Total raw bytes: 3842232975360 (3.49 TB)
          Remaining raw bytes: 146090593065(136.06 GB)
          Used raw bytes: 3240864964620 (2.95 TB)

          - Include the running version of Hadoop.

          - What is the meaning of "Total effective bytes"?

          - Display the hostname instead of the IP address for the data node (toggle option?)
          Jakob Homan made changes -
          Attachment HADOOP-5094.patch [ 12399035 ]
          Jakob Homan made changes -
          Link This issue is related to HADOOP-4281 [ HADOOP-4281 ]
          Jakob Homan made changes -
          Field Original Value New Value
          Assignee Jakob Homan [ jghoman ]
          Jim Huang created issue -

            People

            • Assignee:
              Jakob Homan
              Reporter:
              Jim Huang
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development