[HDFS-9902] Support different values of dfs.datanode.du.reserved per storage type

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.2
• Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
• Component/s: datanode
• Target Version/s: 2.7.4
• Hadoop Flags: Reviewed

Release Note

Reserved space can be configured independently for different storage types for clusters with heterogeneous storage. The 'dfs.datanode.du.reserved' property name can be suffixed with a storage types (i.e. one of ssd, disk, archival or ram_disk). e.g. reserved space for RAM_DISK storage can be configured using the property 'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is not configured then the value specified by 'dfs.datanode.du.reserved' will be used for all volumes.

• Assignee: Brahma Reddy Battula
• Reporter: Pan Yuxuan
• Created: 04/Mar/16 07:43
• Updated: 11/May/17 05:04
• Resolved: 03/May/16 23:56

Description

Now Hadoop support different storage type for DISK, SSD, ARCHIVE and RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
The DISK size may be several TB and the RAM_DISK size may be only several tens of GB.
The problem is that when I configure DISK and RAM_DISK (tmpfs) in the same DN, and I set dfs.datanode.du.reserved values 10GB, this will waste a lot of RAM_DISK size.
Since the usage of RAM_DISK can be 100%, so I don't want dfs.datanode.du.reserved configured for DISK impacts the usage of tmpfs.
So can we make a new configuration for RAM_DISK or just skip this configuration for RAM_DISK?

Issue Links

• relates to: HDFS-11785 Backport HDFS-9902 to branch-2.7: Support different values of dfs.datanode.du.reserved per storage type