[#HADOOP-4281] Capacity reported in some of the commands is not consistent with the Web UI reported data

Hadoop Common

Capacity reported in some of the commands is not consistent with the Web UI reported data

Created: 25/Sep/08 11:02 PM Updated: 08/Jul/09 04:43 PM

Return to search

Component/s:

None

Affects Version/s:

0.19.0

Fix Version/s:

0.19.0

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
HADOOP-4281.patch	2008-09-29 11:04 PM	Suresh Srinivas	4 kB
HADOOP-4281.patch	2008-09-27 05:48 AM	Suresh Srinivas	4 kB

Issue Links:

Reference

This issue relates to:

~~HADOOP-5094~~ Show dead nodes information in dfsadmin -report

~~HADOOP-2816~~ Cluster summary at name node web has confusing report for space utilization

Hadoop Flags:	Reviewed, Incompatible change
Release Note:	« Hide Changed command "hadoop dfsadmin -report" to be consistent with Web UI for both Namenode and Datanode reports. "Total raw bytes" is changed to "Configured Capacity". "Present Capacity" is newly added to indicate the present capacity of the DFS. "Remaining raw bytes" is changed to "DFS Remaining". "Used raw bytes" is changed to "DFS Used". "% used" is changed to "DFS Used%". Applications that parse command output should be reviewed. Show » Changed command "hadoop dfsadmin -report" to be consistent with Web UI for both Namenode and Datanode reports. "Total raw bytes" is changed to "Configured Capacity". "Present Capacity" is newly added to indicate the present capacity of the DFS. "Remaining raw bytes" is changed to "DFS Remaining". "Used raw bytes" is changed to "DFS Used". "% used" is changed to "DFS Used%". Applications that parse command output should be reviewed.
Resolution Date:	30/Sep/08 09:32 PM

Description

« Hide

Changes submitted for 2816 changed the Web UI to report the capacity differently. The same changes need to be applied to commands that show namenode and datanode capacities

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

[ Permlink | « Hide ]

Arun C Murthy added a comment - 25/Sep/08 11:18 PM

Shouldn't this be a blocker for 0.19.0?

[ Permlink | « Hide ]

Suresh Srinivas added a comment - 26/Sep/08 09:17 PM

The change introduces new definition of an item in the report and an additional item that did not exist. This certainly makes Web UI and commands incompatible. Since the report information is not used for building any critical functionality, it should not introduce any major issues.

Hairong, can you please comment?

[ Permlink | « Hide ]

Hairong Kuang added a comment - 26/Sep/08 09:44 PM

I would prefer to make this change to be part of release 19. It's nicer to make the user-facing interfaces to be consistent in one release.

[ Permlink | « Hide ]

Suresh Srinivas added a comment - 27/Sep/08 05:48 AM

All the unit tests passed. Here is the result of ant test-patch. The automated tests for this change was submitted as a part of 2816.

[exec] -1 overall.

[exec] +1 @author. The patch does not contain any @author tags.

[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.

[exec] +1 javadoc. The javadoc tool did not generate any warning messages.

[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.

[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.

[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

Here is the New report with the change compared to the old report
============================================================================================================================
cdev10 04:23:26 ~/hadoop-core-trunk/build/hadoop-0.20.0-dev $ bin/hadoop dfsadmin -report
08/09/27 04:23:48 WARN fs.FileSystem: "localhost:9000" is a deprecated filesystem name. Use "hdfs://localhost:9000/" instead.
08/09/27 04:23:48 WARN fs.FileSystem: "localhost:9000" is a deprecated filesystem name. Use "hdfs://localhost:9000/" instead.
Upgrade for version -18 has been completed.
Upgrade is not finalized.
Configured Capacity: 6339239936 (5.9 GB)
Present Capacity: 3782686528 (3.52 GB)
DFS Remaining: 2781669184 (2.59 GB)
DFS Used: 1001017344 (954.64 MB)
DFS Used%: 26.46%

-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)

Name: 127.0.0.1:50010
Decommission Status : Normal
Configured Capacity: 6339239936 (5.9 GB)
Present Capacity: 3782686528 (3.52 GB)
DFS Remaining: 2781669184(2.59 GB)
DFS Used: 1001017344 (954.64 MB)
DFS Used%: 26.46%
Last contact: Sat Sep 27 04:23:46 UTC 2008

Old report
============================================================================================================================
ucdev10 04:44:22 ~/hadoop-core-trunk/build/hadoop-0.20.0-dev $ bin/hadoop dfsadmin -report
08/09/27 04:44:41 WARN fs.FileSystem: "localhost:9000" is a deprecated filesystem name. Use "hdfs://localhost:9000/" instead.
08/09/27 04:44:41 WARN fs.FileSystem: "localhost:9000" is a deprecated filesystem name. Use "hdfs://localhost:9000/" instead.
Upgrade for version -18 has been completed.
Upgrade is not finalized.
Total raw bytes: 6339239936 (5.9 GB)
Remaining raw bytes: 2781641085 (2.59 GB)
Used raw bytes: 1001017344 (954.64 MB)
% used: 15.79%

-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)

Name: 127.0.0.1:50010
Decommission Status : Normal
Total raw bytes: 6339239936 (5.9 GB)
Remaining raw bytes: 2781641085(2.59 GB)
Used raw bytes: 1001017344 (954.64 MB)
% used: 15.79%
Last contact: Sat Sep 27 04:44:39 UTC 2008
============================================================================================================================

[ Permlink | « Hide ]

Suresh Srinivas added a comment - 29/Sep/08 06:40 PM

Changed the priority to blocker. Without this change the capacity reports from the Web UI and the hadoop dfsadmin -report command is incompatible.

[ Permlink | « Hide ]

Tsz Wo (Nicholas), SZE added a comment - 29/Sep/08 06:58 PM

Should this be a blocker of 0.19?

[ Permlink | « Hide ]

Hairong Kuang added a comment - 29/Sep/08 09:49 PM

The patch looks good. I have two minor comments:
1. In DFSAdmin.java, it would be nice to switch the following two statements
+ long presentCapacity = ds.getDfsUsed() + ds.getRemaining();
+ long used = ds.getDfsUsed();
to be
long used = ds.getDfsUsed();
long presentCapacity = used + ds.getRemaining();
2. In DatanodeInfo.toString, it would be nice to use dfs.getDfsUsedPercent() to calculate the %Used.

I think Suresh meant to mark this issue as a block to 0.19. I will mark it for him.

[ Permlink | « Hide ]

Suresh Srinivas added a comment - 29/Sep/08 11:04 PM

Updated patch based on the comments

[ Permlink | « Hide ]

Hairong Kuang added a comment - 29/Sep/08 11:31 PM

+1. The patch looks good.

[ Permlink | « Hide ]

Hadoop QA added a comment - 30/Sep/08 06:44 AM

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12391171/HADOOP-4281.patch
against trunk revision 700322.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3400/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3400/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3400/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3400/console

This message is automatically generated.

[ Permlink | « Hide ]

Suresh Srinivas added a comment - 30/Sep/08 04:50 PM

HDFS test TestLeaseRecovery2 is failing. It is unrelated to this change (it is failing for other patches too)

[ Permlink | « Hide ]

Chris Douglas added a comment - 30/Sep/08 09:32 PM

I just committed this. Thanks, Suresh

[ Permlink | « Hide ]

Hudson added a comment - 01/Oct/08 01:29 PM

Integrated in Hadoop-trunk #620 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/620/)
. Change dfsadmin to report available disk space in a format
consistent with the web interface as defined in ~~HADOOP-2816~~. Contributed by
Suresh Srinivas