[HDFS-11257] Evacuate DN when the remaining is negative

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: datanode
• Assignee: Anbang Hu
• Reporter: Íñigo Goiri
• Created: 16/Dec/16 19:52
• Updated: 29/Jun/18 20:13

Description

Datanodes have a maximum amount of disk they can use. This is set using dfs.datanode.du.reserved. For example, if we have a 1TB disk and we set the reserved to 100GB, the DN can only use ~900GB. However, if we fill the DN and later other processes (e.g., logs or co-located services) start to use the disk space, the remaining space will go to a negative and the used storage >100%.

The Rebalancer or decommissioning would cover this situation. However, both approaches require administrator intervention while this is a situation that violates the settings. Note that decommisioning, would be too extreme as it would evacuate all the data.

Issue Links

• is related to: HDFS-10285 Storage Policy Satisfier in HDFS