Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8986

Add option to -du to calculate directory space usage excluding snapshots

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Add a -x option for "hdfs -du" and "hdfs -count" commands to exclude snapshots from being calculated.

      Description

      When running hadoop fs -du on a snapshotted directory (or one of its children), the report includes space consumed by blocks that are only present in the snapshots. This is confusing for end users.

      $  hadoop fs -du -h -s /tmp/parent /tmp/parent/*
      799.7 M  2.3 G  /tmp/parent
      799.7 M  2.3 G  /tmp/parent/sub1
      
      $ hdfs dfs -createSnapshot /tmp/parent snap1
      Created snapshot /tmp/parent/.snapshot/snap1
      
      $ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
      ...
      
      $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
      799.7 M  2.3 G  /tmp/parent
      799.7 M  2.3 G  /tmp/parent/sub1
      
      $ hdfs dfs -deleteSnapshot /tmp/parent snap1
      $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
      0  0  /tmp/parent
      0  0  /tmp/parent/sub1
      

      It would be helpful if we had a flag, say -X, to exclude any snapshot related disk usage in the output

        Attachments

        1. HDFS-8986.01.patch
          34 kB
          Xiao Chen
        2. HDFS-8986.02.patch
          38 kB
          Xiao Chen
        3. HDFS-8986.03.patch
          39 kB
          Xiao Chen
        4. HDFS-8986.04.patch
          39 kB
          Xiao Chen
        5. HDFS-8986.05.patch
          39 kB
          Xiao Chen
        6. HDFS-8986.06.patch
          39 kB
          Wei-Chiu Chuang
        7. HDFS-8986.07.patch
          40 kB
          Xiao Chen
        8. HDFS-8986.branch-2.patch
          39 kB
          Xiao Chen

          Issue Links

            Activity

              People

              • Assignee:
                xiaochen Xiao Chen
                Reporter:
                ggop Gautam Gopalakrishnan
              • Votes:
                1 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: