Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8986

Add option to -du to calculate directory space usage excluding snapshots

    XMLWordPrintableJSON

Details

    • Reviewed
    • Add a -x option for "hdfs -du" and "hdfs -count" commands to exclude snapshots from being calculated.

    Description

      When running hadoop fs -du on a snapshotted directory (or one of its children), the report includes space consumed by blocks that are only present in the snapshots. This is confusing for end users.

      $  hadoop fs -du -h -s /tmp/parent /tmp/parent/*
      799.7 M  2.3 G  /tmp/parent
      799.7 M  2.3 G  /tmp/parent/sub1
      
      $ hdfs dfs -createSnapshot /tmp/parent snap1
      Created snapshot /tmp/parent/.snapshot/snap1
      
      $ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
      ...
      
      $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
      799.7 M  2.3 G  /tmp/parent
      799.7 M  2.3 G  /tmp/parent/sub1
      
      $ hdfs dfs -deleteSnapshot /tmp/parent snap1
      $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
      0  0  /tmp/parent
      0  0  /tmp/parent/sub1
      

      It would be helpful if we had a flag, say -X, to exclude any snapshot related disk usage in the output

      Attachments

        1. HDFS-8986.branch-2.patch
          39 kB
          Xiao Chen
        2. HDFS-8986.07.patch
          40 kB
          Xiao Chen
        3. HDFS-8986.06.patch
          39 kB
          Wei-Chiu Chuang
        4. HDFS-8986.05.patch
          39 kB
          Xiao Chen
        5. HDFS-8986.04.patch
          39 kB
          Xiao Chen
        6. HDFS-8986.03.patch
          39 kB
          Xiao Chen
        7. HDFS-8986.02.patch
          38 kB
          Xiao Chen
        8. HDFS-8986.01.patch
          34 kB
          Xiao Chen

        Issue Links

          Activity

            People

              xiaochen Xiao Chen
              ggop Gautam Gopalakrishnan
              Votes:
              1 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: