Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11058

Implement 'hadoop fs -df' command for ViewFileSystem

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: None
    • Labels:

      Description

      Df command doesn't seem to work well with ViewFileSystem. It always reports used data as 0. Here is the client mount table configuration I am using against a federated clusters of 2 NameNodes and 2 DataNoes.

        1 <?xml version="1.0" ?>
        2 <configuration>
        3   <property>
        4     <name>fs.defaultFS</name>
        5     <value>viewfs://ClusterX/</value>
        6   </property>
        ..
       11   <property>
       12     <name>fs.default.name</name>
       13     <value>viewfs://ClusterX/</value>
       14   </property>
       ..
       23   <property>
       24     <name>fs.viewfs.mounttable.ClusterX.link./nn0</name>
       25     <value>hdfs://127.0.0.1:50001/</value>
       26   </property>
       27   <property>
       28     <name>fs.viewfs.mounttable.ClusterX.link./nn1</name>
       29     <value>hdfs://127.0.0.1:51001/</value>
       30   </property>
       31   <property>
       32     <name>fs.viewfs.mounttable.ClusterX.link./nn2</name>
       33     <value>hdfs://127.0.0.1:52001/nn2</value>
       34   </property>
       35   <property>
       36     <name>fs.viewfs.mounttable.ClusterX.link./nn3</name>
       37     <value>hdfs://127.0.0.1:52001/nn3</value>
       38   </property>
       39   <property>
       40     <name>fs.viewfs.mounttable.ClusterY.linkMergeSlash</name>
       41     <value>hdfs://127.0.0.1:50001/</value>
       42   </property>
       43 </configuration>
      

      Df command always reports Size/Available as 8.0E and the usage as 0 for any federated cluster.

      # hadoop fs -fs viewfs://ClusterX/ -df  /
      Filesystem                         Size  Used            Available  Use%
      viewfs://ClusterX/  9223372036854775807     0  9223372036854775807    0%
      
      # hadoop fs -fs viewfs://ClusterX/ -df  -h /
      Filesystem           Size  Used  Available  Use%
      viewfs://ClusterX/  8.0 E     0      8.0 E    0%
      
      # hadoop fs -fs viewfs://ClusterY/ -df  -h /
      Filesystem           Size  Used  Available  Use%
      viewfs://ClusterY/  8.0 E     0      8.0 E    0%
      

      Whereas Du command seems to work as expected even with ViewFileSystem.

      # hadoop fs -fs viewfs://ClusterY/ -du -h /
      10.6 K  31.8 K  /build.log.16y
      0       0       /user
      
      # hadoop fs -fs viewfs://ClusterX/ -du -h /
      10.6 K  31.8 K  /nn0
      0       0       /nn1
      20.2 K  35.8 K  /nn3
      40.6 K  34.3 K  /nn4
      
      

        Attachments

        1. HDFS-11058.04.patch
          27 kB
          Manoj Govindassamy
        2. HDFS-11058.03.patch
          27 kB
          Manoj Govindassamy
        3. HDFS-11058.02.patch
          30 kB
          Manoj Govindassamy
        4. HDFS-11058.01.patch
          31 kB
          Manoj Govindassamy

          Issue Links

            Activity

              People

              • Assignee:
                manojg Manoj Govindassamy
                Reporter:
                manojg Manoj Govindassamy
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: