Hadoop Common
  1. Hadoop Common
  2. HADOOP-4281

Capacity reported in some of the commands is not consistent with the Web UI reported data

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Changed command "hadoop dfsadmin -report" to be consistent with Web UI for both Namenode and Datanode reports. "Total raw bytes" is changed to "Configured Capacity". "Present Capacity" is newly added to indicate the present capacity of the DFS. "Remaining raw bytes" is changed to "DFS Remaining". "Used raw bytes" is changed to "DFS Used". "% used" is changed to "DFS Used%". Applications that parse command output should be reviewed.
      Show
      Changed command "hadoop dfsadmin -report" to be consistent with Web UI for both Namenode and Datanode reports. "Total raw bytes" is changed to "Configured Capacity". "Present Capacity" is newly added to indicate the present capacity of the DFS. "Remaining raw bytes" is changed to "DFS Remaining". "Used raw bytes" is changed to "DFS Used". "% used" is changed to "DFS Used%". Applications that parse command output should be reviewed.

      Description

      Changes submitted for 2816 changed the Web UI to report the capacity differently. The same changes need to be applied to commands that show namenode and datanode capacities

      1. HADOOP-4281.patch
        4 kB
        Suresh Srinivas
      2. HADOOP-4281.patch
        4 kB
        Suresh Srinivas

        Issue Links

          Activity

          Hide
          Arun C Murthy added a comment -

          Shouldn't this be a blocker for 0.19.0?

          Show
          Arun C Murthy added a comment - Shouldn't this be a blocker for 0.19.0?
          Hide
          Suresh Srinivas added a comment -

          The change introduces new definition of an item in the report and an additional item that did not exist. This certainly makes Web UI and commands incompatible. Since the report information is not used for building any critical functionality, it should not introduce any major issues.

          Hairong, can you please comment?

          Show
          Suresh Srinivas added a comment - The change introduces new definition of an item in the report and an additional item that did not exist. This certainly makes Web UI and commands incompatible. Since the report information is not used for building any critical functionality, it should not introduce any major issues. Hairong, can you please comment?
          Hide
          Hairong Kuang added a comment -

          I would prefer to make this change to be part of release 19. It's nicer to make the user-facing interfaces to be consistent in one release.

          Show
          Hairong Kuang added a comment - I would prefer to make this change to be part of release 19. It's nicer to make the user-facing interfaces to be consistent in one release.
          Hide
          Suresh Srinivas added a comment -

          All the unit tests passed. Here is the result of ant test-patch. The automated tests for this change was submitted as a part of 2816.

          [exec] -1 overall.

          [exec] +1 @author. The patch does not contain any @author tags.

          [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
          [exec] Please justify why no tests are needed for this patch.

          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.

          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.

          [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          Here is the New report with the change compared to the old report
          ============================================================================================================================
          cdev10 04:23:26 ~/hadoop-core-trunk/build/hadoop-0.20.0-dev $ bin/hadoop dfsadmin -report
          08/09/27 04:23:48 WARN fs.FileSystem: "localhost:9000" is a deprecated filesystem name. Use "hdfs://localhost:9000/" instead.
          08/09/27 04:23:48 WARN fs.FileSystem: "localhost:9000" is a deprecated filesystem name. Use "hdfs://localhost:9000/" instead.
          Upgrade for version -18 has been completed.
          Upgrade is not finalized.
          Configured Capacity: 6339239936 (5.9 GB)
          Present Capacity: 3782686528 (3.52 GB)
          DFS Remaining: 2781669184 (2.59 GB)
          DFS Used: 1001017344 (954.64 MB)
          DFS Used%: 26.46%

          -------------------------------------------------
          Datanodes available: 1 (1 total, 0 dead)

          Name: 127.0.0.1:50010
          Decommission Status : Normal
          Configured Capacity: 6339239936 (5.9 GB)
          Present Capacity: 3782686528 (3.52 GB)
          DFS Remaining: 2781669184(2.59 GB)
          DFS Used: 1001017344 (954.64 MB)
          DFS Used%: 26.46%
          Last contact: Sat Sep 27 04:23:46 UTC 2008

          Old report
          ============================================================================================================================
          ucdev10 04:44:22 ~/hadoop-core-trunk/build/hadoop-0.20.0-dev $ bin/hadoop dfsadmin -report
          08/09/27 04:44:41 WARN fs.FileSystem: "localhost:9000" is a deprecated filesystem name. Use "hdfs://localhost:9000/" instead.
          08/09/27 04:44:41 WARN fs.FileSystem: "localhost:9000" is a deprecated filesystem name. Use "hdfs://localhost:9000/" instead.
          Upgrade for version -18 has been completed.
          Upgrade is not finalized.
          Total raw bytes: 6339239936 (5.9 GB)
          Remaining raw bytes: 2781641085 (2.59 GB)
          Used raw bytes: 1001017344 (954.64 MB)
          % used: 15.79%

          -------------------------------------------------
          Datanodes available: 1 (1 total, 0 dead)

          Name: 127.0.0.1:50010
          Decommission Status : Normal
          Total raw bytes: 6339239936 (5.9 GB)
          Remaining raw bytes: 2781641085(2.59 GB)
          Used raw bytes: 1001017344 (954.64 MB)
          % used: 15.79%
          Last contact: Sat Sep 27 04:44:39 UTC 2008
          ============================================================================================================================

          Show
          Suresh Srinivas added a comment - All the unit tests passed. Here is the result of ant test-patch. The automated tests for this change was submitted as a part of 2816. [exec] -1 overall. [exec] +1 @author. The patch does not contain any @author tags. [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. Here is the New report with the change compared to the old report ============================================================================================================================ cdev10 04:23:26 ~/hadoop-core-trunk/build/hadoop-0.20.0-dev $ bin/hadoop dfsadmin -report 08/09/27 04:23:48 WARN fs.FileSystem: "localhost:9000" is a deprecated filesystem name. Use "hdfs://localhost:9000/" instead. 08/09/27 04:23:48 WARN fs.FileSystem: "localhost:9000" is a deprecated filesystem name. Use "hdfs://localhost:9000/" instead. Upgrade for version -18 has been completed. Upgrade is not finalized. Configured Capacity: 6339239936 (5.9 GB) Present Capacity: 3782686528 (3.52 GB) DFS Remaining: 2781669184 (2.59 GB) DFS Used: 1001017344 (954.64 MB) DFS Used%: 26.46% ------------------------------------------------- Datanodes available: 1 (1 total, 0 dead) Name: 127.0.0.1:50010 Decommission Status : Normal Configured Capacity: 6339239936 (5.9 GB) Present Capacity: 3782686528 (3.52 GB) DFS Remaining: 2781669184(2.59 GB) DFS Used: 1001017344 (954.64 MB) DFS Used%: 26.46% Last contact: Sat Sep 27 04:23:46 UTC 2008 Old report ============================================================================================================================ ucdev10 04:44:22 ~/hadoop-core-trunk/build/hadoop-0.20.0-dev $ bin/hadoop dfsadmin -report 08/09/27 04:44:41 WARN fs.FileSystem: "localhost:9000" is a deprecated filesystem name. Use "hdfs://localhost:9000/" instead. 08/09/27 04:44:41 WARN fs.FileSystem: "localhost:9000" is a deprecated filesystem name. Use "hdfs://localhost:9000/" instead. Upgrade for version -18 has been completed. Upgrade is not finalized. Total raw bytes: 6339239936 (5.9 GB) Remaining raw bytes: 2781641085 (2.59 GB) Used raw bytes: 1001017344 (954.64 MB) % used: 15.79% ------------------------------------------------- Datanodes available: 1 (1 total, 0 dead) Name: 127.0.0.1:50010 Decommission Status : Normal Total raw bytes: 6339239936 (5.9 GB) Remaining raw bytes: 2781641085(2.59 GB) Used raw bytes: 1001017344 (954.64 MB) % used: 15.79% Last contact: Sat Sep 27 04:44:39 UTC 2008 ============================================================================================================================
          Hide
          Suresh Srinivas added a comment -

          Changed the priority to blocker. Without this change the capacity reports from the Web UI and the hadoop dfsadmin -report command is incompatible.

          Show
          Suresh Srinivas added a comment - Changed the priority to blocker. Without this change the capacity reports from the Web UI and the hadoop dfsadmin -report command is incompatible.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Should this be a blocker of 0.19?

          Show
          Tsz Wo Nicholas Sze added a comment - Should this be a blocker of 0.19?
          Hide
          Hairong Kuang added a comment -

          The patch looks good. I have two minor comments:
          1. In DFSAdmin.java, it would be nice to switch the following two statements
          + long presentCapacity = ds.getDfsUsed() + ds.getRemaining();
          + long used = ds.getDfsUsed();
          to be
          long used = ds.getDfsUsed();
          long presentCapacity = used + ds.getRemaining();
          2. In DatanodeInfo.toString, it would be nice to use dfs.getDfsUsedPercent() to calculate the %Used.

          I think Suresh meant to mark this issue as a block to 0.19. I will mark it for him.

          Show
          Hairong Kuang added a comment - The patch looks good. I have two minor comments: 1. In DFSAdmin.java, it would be nice to switch the following two statements + long presentCapacity = ds.getDfsUsed() + ds.getRemaining(); + long used = ds.getDfsUsed(); to be long used = ds.getDfsUsed(); long presentCapacity = used + ds.getRemaining(); 2. In DatanodeInfo.toString, it would be nice to use dfs.getDfsUsedPercent() to calculate the %Used. I think Suresh meant to mark this issue as a block to 0.19. I will mark it for him.
          Hide
          Suresh Srinivas added a comment -

          Updated patch based on the comments

          Show
          Suresh Srinivas added a comment - Updated patch based on the comments
          Hide
          Hairong Kuang added a comment -

          +1. The patch looks good.

          Show
          Hairong Kuang added a comment - +1. The patch looks good.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12391171/HADOOP-4281.patch
          against trunk revision 700322.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3400/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3400/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3400/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3400/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12391171/HADOOP-4281.patch against trunk revision 700322. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3400/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3400/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3400/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3400/console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          HDFS test TestLeaseRecovery2 is failing. It is unrelated to this change (it is failing for other patches too)

          Show
          Suresh Srinivas added a comment - HDFS test TestLeaseRecovery2 is failing. It is unrelated to this change (it is failing for other patches too)
          Hide
          Chris Douglas added a comment -

          I just committed this. Thanks, Suresh

          Show
          Chris Douglas added a comment - I just committed this. Thanks, Suresh
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk #620 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/620/)
          . Change dfsadmin to report available disk space in a format
          consistent with the web interface as defined in HADOOP-2816. Contributed by
          Suresh Srinivas

          Show
          Hudson added a comment - Integrated in Hadoop-trunk #620 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/620/ ) . Change dfsadmin to report available disk space in a format consistent with the web interface as defined in HADOOP-2816 . Contributed by Suresh Srinivas

            People

            • Assignee:
              Suresh Srinivas
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development