[HDFS-1312] Re-balance disks within a Datanode

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha1
• Component/s: datanode
• Hadoop Flags: Reviewed

Release Note

The Disk Balancer lets administrators rebalance data across multiple disks of a DataNode. It is useful to correct skewed data distribution often seen after adding or replacing disks. Disk Balancer can be enabled by setting dfs.disk.balancer.enabled to true in hdfs-site.xml. It can be invoked by running "hdfs diskbalancer". See the "HDFS Diskbalancer" section in the HDFS Commands guide for detailed usage.

• Assignee: Anu Engineer
• Reporter: Travis Crawford
• Created: 21/Jul/10 23:46
• Updated: 18/Jul/16 21:11
• Resolved: 24/Jun/16 21:19

Description

Filing this issue in response to ``full disk woes`` on hdfs-user.

Datanodes fill their storage directories unevenly, leading to situations where certain disks are full while others are significantly less used. Users at many different sites have experienced this issue, and HDFS administrators are taking steps like:

• Manually rebalancing blocks in storage directories
• Decomissioning nodes & later readding them

There's a tradeoff between making use of all available spindles, and filling disks at the sameish rate. Possible solutions include:

• Weighting less-used disks heavier when placing new blocks on the datanode. In write-heavy environments this will still make use of all spindles, equalizing disk use over time.
• Rebalancing blocks locally. This would help equalize disk use as disks are added/replaced in older cluster nodes.

Datanodes should actively manage their local disk so operator intervention is not needed.

Issue Links

• is duplicated by: HDFS-4408 Balancer for DataNode's Volumes
• is related to: HDFS-10576 DiskBalancer followup work items
• is related to: HDFS-1804 Add a new block-volume device choosing policy that looks at free space
• is related to: HDFS-1121 Allow HDFS client to measure distribution of blocks across devices for a specific DataNode
• is related to: HDFS-1120 Make DataNode's block-to-device placement policy pluggable
• relates to: HDFS-8538 Change the default volume choosing policy to AvailableSpaceVolumeChoosingPolicy