[HDFS-11191] Datanode Capacity is misleading if the dfs.datanode.data.dir is configured with two directories from the same file system.

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.5.0
• Component/s: hdfs
• Labels: capacity datanode storage user-experience

Environment

SLES 11SP3
HDP 2.5.0

• Assignee: Weiwei Yang
• Reporter: Deepak Chander
• Created: 30/Nov/16 07:28
• Updated: 11/Jan/17 03:49

Description

In the command “hdfs dfsadmin -report” The Configured Capacity is misleading if the dfs.datanode.data.dir is configured with two directories from the same file system.

hdfs@kimtest1:~> hdfs dfsadmin -report
Configured Capacity: 239942369274 (223.46 GB)
Present Capacity: 207894724602 (193.62 GB)
DFS Remaining: 207894552570 (193.62 GB)
DFS Used: 172032 (168 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (3):

Name: 172.26.79.87:50010 (kimtest3)
Hostname: kimtest3
Decommission Status : Normal
Configured Capacity: 79980789758 (74.49 GB)
DFS Used: 57344 (56 KB)
Non DFS Used: 9528000512 (8.87 GB)
DFS Remaining: 70452731902 (65.61 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.09%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 06:59:02 PST 2016

Name: 172.26.80.38:50010 (kimtest4)
Hostname: kimtest4
Decommission Status : Normal
Configured Capacity: 79980789758 (74.49 GB)
DFS Used: 57344 (56 KB)
Non DFS Used: 13010952192 (12.12 GB)
DFS Remaining: 66969780222 (62.37 GB)
DFS Used%: 0.00%
DFS Remaining%: 83.73%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 06:59:02 PST 2016

Name: 172.26.79.86:50010 (kimtest2)
Hostname: kimtest2
Decommission Status : Normal
Configured Capacity: 79980789758 (74.49 GB)
DFS Used: 57344 (56 KB)
Non DFS Used: 9508691968 (8.86 GB)
DFS Remaining: 70472040446 (65.63 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.11%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 06:59:02 PST 2016

If you see my datanode root file system size its only 38GB

kimtest3:~ # df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 38G 2.6G 33G 8% /

kimtest4:~ # df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 38G 4.2G 32G 12% /

kimtest2:~ # df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 38G 2.6G 33G 8% /

The below is from hdfs-site.xml file

<property>
<name>dfs.datanode.data.dir</name>
<value>file:///grid/hadoop/hdfs/dn, file:///grid1/hadoop/hdfs/dn</value>
</property>

I have removed the other directory grid1 and restarted datanode process.

<property>
<name>dfs.datanode.data.dir</name>
<value>file:///grid/hadoop/hdfs/dn</value>
</property>

Now the size is reflecting correctly

hdfs@kimtest1:/grid> hdfs dfsadmin -report
Configured Capacity: 119971184637 (111.73 GB)
Present Capacity: 103947243517 (96.81 GB)
DFS Remaining: 103947157501 (96.81 GB)
DFS Used: 86016 (84 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (3):

Name: 172.26.79.87:50010 (kimtest3)
Hostname: kimtest3
Decommission Status : Normal
Configured Capacity: 39990394879 (37.24 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 4764057600 (4.44 GB)
DFS Remaining: 35226308607 (32.81 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.09%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 07:34:02 PST 2016

Name: 172.26.80.38:50010 (kimtest4)
Hostname: kimtest4
Decommission Status : Normal
Configured Capacity: 39990394879 (37.24 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 6505525248 (6.06 GB)
DFS Remaining: 33484840959 (31.19 GB)
DFS Used%: 0.00%
DFS Remaining%: 83.73%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 07:34:01 PST 2016

Name: 172.26.79.86:50010 (kimtest2)
Hostname: kimtest2
Decommission Status : Normal
Configured Capacity: 39990394879 (37.24 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 4754358272 (4.43 GB)
DFS Remaining: 35236007935 (32.82 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.11%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 07:34:02 PST 2016

Issue Links

• duplicates: HDFS-8610 if set several dirs which belongs to one disk in "dfs.datanode.data.dir", NN calculate capacity wrong