|
For reporting the following info needs to be considered:
Total capacity - Capacity of all the data directories Here is how DFS remaining space is calculated: Current proposal does not consider the factor dfs.datanode.du.pct. I am not sure why du.pct is being used. If it is to reduce available disk space for DFS, to consider factors such as disk fragmentation - it is not serving the purpose. Available space keeps on decreasing. The percentage is applied to the shrinking available space. Eventually the DFS ends up using all the available space any way (in theory) and the du.pct will not serve any purpose. My proposal: or 2) If du.pct is used, it is calculated on Total capacity and not on available space. This helps set aside a percentage of total capacity. After discussing this with Hairong, looks like the issues of dfs.datanode.du.pct is unrelated to reporting data. The du.pct issue will be tracked in a separate JIRA.
The data displayed is changed as follows: Cluster Summary Present Capacity: This is newly added and represents the present capacity available for DFS use. This is sum of DFS Remaining and DFS Used given below DFS Remaining : This will remain as it is Node data prints currently: It will be change to: Size column is renamed as Capacity. Previously this was calculated as sum of file system capacity of all the data directories. It is changed to exclude reserved space and will be calculated as (sum of file system capacity of all the data directories - reserved space) New column Present Capacity is added. This will sum of Used and Remaining. Attached file makes the proposed changes. One change from my previous comment is, the used percentages are calculated based on the Present Capacity instead of Total Capacity.
1. FSDataSet.java: getCapacity() should make sure that it does not return a negative number.
2. FSNamesystem.java: In getCapacityUsedPercent(), used space should be divided by the present capacity. FSNamesystem probably should not have this public method since it is only used in a test. 3. DatanodeInfo.getDfsUsedPercent should check the case that the present capacity is zero. Again, I do not think this public method needs to add to the class since it is only used in webUI and the test. 4. In webUI, better to rename "Total Capacity" to be "Configured Capacity" to show that it is different from the old definition. 5. Since the capacity field in the heartbeat has a new definition, should we bump up the DatanodeProtocol version? Suresh, could you please also change the command line cluster report? This is an extra work, but I think it is better to make the command line report and web UI report to be consistent in one release. Please take a look at DFSAdmin.report() and DatanodeInfo.getDatanodeReport(). Thanks.
Thanks for the review. I have uploaded new patch with the changes.
1. FSDataSet.java: getCapacity() should make sure that it does not return a negative number. 2. FSNamesystem.java: In getCapacityUsedPercent(), used space should be divided by the present capacity. FSNamesystem probably should not have this public method since it is only used in a test. 3. DatanodeInfo.getDfsUsedPercent should check the case that the present capacity is zero. Again, I do not think this public method needs to add to the class since it is only used in webUI and the test. 4. In webUI, better to rename "Total Capacity" to be "Configured Capacity" to show that it is different from the old definition. 5. Since the capacity field in the heartbeat has a new definition, should we bump up the DatanodeProtocol version? Additionally a new JIRA will be created to keep track of reporting the capacity for DFSAdmin report and other CLIs that are impacted by this change. This change only addresses the Web UI. -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12390442/HADOOP-2816.patch  against trunk revision 696846. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3312/testReport/ This message is automatically generated. New patch passed all the unit tests.
Test results for the test-patch: [exec] +1 @author. The patch does not contain any @author tags. [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. I've committed this. Thanks, Suresh!
Integrated in Hadoop-trunk #611 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/611/
) Integrated in Hadoop-trunk #620 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/620/
)consistent with the web interface as defined in . Contributed by Suresh Srinivas Changes are made as described in the proposed solution (in the previous comment).
Here is the test-patch result: [exec] +1 @author. The patch does not contain any @author tags. [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. Ignore the previous comment. It was intended for another issue.
This fix changes the following:
1) Datanode heartbeat reported Capacity information is changed. Earlier the Capacity was sum of all the diskspace of data directories. With this change, it is sum of all the diskspace of data directories minus the reserved space configured using dfs.datanode.du.reserved config param. This change is reflected by changing the protocol version from 17 to 18. 2) The Namenode Web UI is changed accordingly as detailed below... Cluster Summary Present Capacity: This is newly added and represents the present capacity available for DFS use. This is sum of DFS Remaining and DFS Used given below DFS Remaining : This will remain as it is Node data prints currently: It will be change to: Size column is renamed as Total Capacity. Previously this was calculated as sum of file system capacity of all the data directories. It is changed to exclude reserved space and will be calculated as (sum of file system capacity of all the data directories - reserved space) |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Improvement
Closed
Major
Cluster summary at name node web has confusing report for space utilization
Allocation and management algorithms will not change, but reporting on the Name Node home page should be modified:
In the node table, four statistics will be reported:
"Configured Capacity" is the sum over all volumes V named in config:dfs.data.dir that exist of (df.Size V - config:dfs.datanode.du.reserved)
"Present Capacity" is df.Size V - MAX{df.Used V - [space used to store block and metadata files], config:dfs.datanode.du.reserved)
"Used (%)" is the ratio of [space used to store block and metadata files] and Present Capacity (the little gauge reflects this value)
"Remaining" is the difference Present Capacity - [space used to store block and metadata files]
The "cluster summary" will report 5 statistics:
"Configured Capacity" is the sum over all data nodes D of d.ConfiguredCapacity
"Present Capacity" is the sum over all data nodes of D of D.PresentCapacity
"Used" is the sum over all data nodes of D of D.[space used to store block and metadata files]
"Remaining" is the difference Present Capacity - Used
"Used %" is the ratio of Used to Present Capacity
A key will explain these calculations for the user.