[HDFS-11058] Implement 'hadoop fs -df' command for ViewFileSystem

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 3.0.0-alpha2
• Labels: viewfs
• Target Version/s: 3.0.0-alpha2
• Assignee: Manoj Govindassamy
• Reporter: Manoj Govindassamy
• Created: 26/Oct/16 00:04
• Updated: 24/Nov/16 00:41
• Resolved: 24/Nov/16 00:41

Description

Df command doesn't seem to work well with ViewFileSystem. It always reports used data as 0. Here is the client mount table configuration I am using against a federated clusters of 2 NameNodes and 2 DataNoes.

  1 <?xml version="1.0" ?>
  2 <configuration>
  3   <property>
  4     <name>fs.defaultFS</name>
  5     <value>viewfs://ClusterX/</value>
  6   </property>
  ..
 11   <property>
 12     <name>fs.default.name</name>
 13     <value>viewfs://ClusterX/</value>
 14   </property>
 ..
 23   <property>
 24     <name>fs.viewfs.mounttable.ClusterX.link./nn0</name>
 25     <value>hdfs://127.0.0.1:50001/</value>
 26   </property>
 27   <property>
 28     <name>fs.viewfs.mounttable.ClusterX.link./nn1</name>
 29     <value>hdfs://127.0.0.1:51001/</value>
 30   </property>
 31   <property>
 32     <name>fs.viewfs.mounttable.ClusterX.link./nn2</name>
 33     <value>hdfs://127.0.0.1:52001/nn2</value>
 34   </property>
 35   <property>
 36     <name>fs.viewfs.mounttable.ClusterX.link./nn3</name>
 37     <value>hdfs://127.0.0.1:52001/nn3</value>
 38   </property>
 39   <property>
 40     <name>fs.viewfs.mounttable.ClusterY.linkMergeSlash</name>
 41     <value>hdfs://127.0.0.1:50001/</value>
 42   </property>
 43 </configuration>

Df command always reports Size/Available as 8.0E and the usage as 0 for any federated cluster.

# hadoop fs -fs viewfs://ClusterX/ -df  /
Filesystem                         Size  Used            Available  Use%
viewfs://ClusterX/  9223372036854775807     0  9223372036854775807    0%

# hadoop fs -fs viewfs://ClusterX/ -df  -h /
Filesystem           Size  Used  Available  Use%
viewfs://ClusterX/  8.0 E     0      8.0 E    0%

# hadoop fs -fs viewfs://ClusterY/ -df  -h /
Filesystem           Size  Used  Available  Use%
viewfs://ClusterY/  8.0 E     0      8.0 E    0%

Whereas Du command seems to work as expected even with ViewFileSystem.

# hadoop fs -fs viewfs://ClusterY/ -du -h /
10.6 K  31.8 K  /build.log.16y
0       0       /user

# hadoop fs -fs viewfs://ClusterX/ -du -h /
10.6 K  31.8 K  /nn0
0       0       /nn1
20.2 K  35.8 K  /nn3
40.6 K  34.3 K  /nn4

Issue Links

• is depended upon by: HDFS-11109 ViewFileSystem Df command should work even when the backing NameServices are down
• is duplicated by: HDFS-11057 Implement 'df' for viewfs