[HADOOP-4430] Namenode Web UI capacity report is inconsistent with Balancer

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.19.0
• Fix Version/s: 0.19.0
• Hadoop Flags: Incompatible change, Reviewed

Release Note

Changed reporting in the NameNode Web UI to more closely reflect the behavior of the re-balancer. Removed no longer used config parameter dfs.datanode.du.pct from hadoop-default.xml.

• Assignee: Suresh Srinivas
• Reporter: Suresh Srinivas
• Created: 16/Oct/08 17:56
• Updated: 06/Jan/11 09:07
• Resolved: 20/Oct/08 18:17

Description

Solution to 2816 changed

• Total Capacity definition from (the disk space of all data directories) to (the disk space of all the data directories - the reserved space)
• We added a new element Present Capacity to the report. It is set to (Used Capacity + Remaining Capacity)
• We changed the Used Percentage reported from (Used Capacity)/(Total Capacity) to (Used Capacity)/(Present Capacity)
• All these changes are displayed on Namenode Web UI.

Balancer functionality
Balancer script is started with a threshold parameter. It tries to move the blocks from the nodes that have Used % that is more than (Cluster average + threshold) to the nodes that have less than (Cluster average - threshold). Essentially balancer gets all the datanodes used % to with in (the Cluster average +/- threshold).

Inconsistencies due to the change in 2816
When MapReduce jobs are run, temporary files are generated. This eats away a lot of space from Present Capacity. The difference between the Total Capacity and the Present Capacity can be huge. Currently balancer computes Used Percentage based (Used Capacity)/(Total Capacity). The Used % the balancer uses could be significantly different from Used % displayed on the Namenode Web UI. When balancer is done balancing, the Namenode Used % might still appear unbalanced.

Issue Links

• blocks: HADOOP-2816 Cluster summary at name node web has confusing report for space utilization
• relates to: HDFS-1564 Make dfs.datanode.du.reserved configurable per volume