Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9063

Correctly handle snapshot path for getContentSummary

    Details

    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      The jira made the following changes:
      1. Fix a bug to exclude newly-created files from quota usage calculation for a snapshot path.
      2. Number of snapshots is no longer counted as directory number in getContentSummary result.
      Show
      The jira made the following changes: 1. Fix a bug to exclude newly-created files from quota usage calculation for a snapshot path. 2. Number of snapshots is no longer counted as directory number in getContentSummary result.

      Description

      The current getContentSummary implementation does not take into account the snapshot path, thus if we have the following ops:
      1. create dirs /foo/bar
      2. take snapshot s1 on /foo
      3. create a 1 byte file /foo/bar/baz

      then "du /foo" and "du /foo/.snapshot/s1" can report same results for "bar", which is incorrect since the 1 byte file is not included in snapshot s1.

      In the meanwhile, the snapshot diff list size is no longer included in the computation result. This can bring minor incompatibility but is consistent with the change in HDFS-7728.

      1. HDFS-9063.000.patch
        19 kB
        Jing Zhao
      2. test.001.patch
        3 kB
        Manoj Govindassamy
      3. test.002.patch
        4 kB
        Manoj Govindassamy

        Activity

        Hide
        jingzhao Jing Zhao added a comment -

        Upload a patch to fix the issue. In the patch I add an extra parameter for INode#computeContentSummary to indicate the snapshot id of the path. In the meanwhile, the snapshot diff list size is no longer included in the computation result. This can bring minor incompatibility but is consistent with the change in HDFS-7728.

        Show
        jingzhao Jing Zhao added a comment - Upload a patch to fix the issue. In the patch I add an extra parameter for INode#computeContentSummary to indicate the snapshot id of the path. In the meanwhile, the snapshot diff list size is no longer included in the computation result. This can bring minor incompatibility but is consistent with the change in HDFS-7728 .
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 26s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 42s There were no new javac warning messages.
        +1 javadoc 10m 53s There were no new javadoc warning messages.
        +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 30s The applied patch generated 2 new checkstyle issues (total was 177, now 179).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 39s mvn install still works.
        +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
        +1 findbugs 2m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 30s Pre-build of native portion
        -1 hdfs tests 74m 57s Tests failed in hadoop-hdfs.
            124m 27s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing
          hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
          hadoop.hdfs.server.namenode.snapshot.TestGetContentSummaryWithSnapshot
          hadoop.hdfs.server.namenode.TestFSNamesystem
        Timed out tests org.apache.hadoop.hdfs.TestRollingUpgrade



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12756374/HDFS-9063.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 941001b
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12496/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12496/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12496/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12496/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 26s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 42s There were no new javac warning messages. +1 javadoc 10m 53s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 30s The applied patch generated 2 new checkstyle issues (total was 177, now 179). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 2m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 30s Pre-build of native portion -1 hdfs tests 74m 57s Tests failed in hadoop-hdfs.     124m 27s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport   hadoop.hdfs.server.namenode.snapshot.TestGetContentSummaryWithSnapshot   hadoop.hdfs.server.namenode.TestFSNamesystem Timed out tests org.apache.hadoop.hdfs.TestRollingUpgrade Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12756374/HDFS-9063.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 941001b checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12496/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12496/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12496/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12496/console This message was automatically generated.
        Hide
        hitliuyi Yi Liu added a comment -

        Thanks Jing Zhao for working on this. Jenkins' report has issue, TestGetContentSummaryWithSnapshot can pass locally.

        I also found similar issue (not the same, but also problem of getContentSummary if there is snapshot) when writing tests for large directory in HDFS-9053, it also exists in current trunk. I apply your patch, the issue I saw is still there.
        I think you can fix the issue too and write test of following steps to reproduce the issue I found (Of course if you don't want to fix it, I can do it separately ):

        1. Suppose we have a directory named 'dir', create 16 files in the dir
        2. remove the last 1 file – now total 15 files in dir
        3. create a snapshot 's1' of dir
        4. add 1 file in dir – now total 16 files in dir
        5. remove the first 1 file in dir – now total 15 files in the dir
        6. call getContentSummary(dir), and then getFileCount. – the expected result is 15, but the return is 16.
        Show
        hitliuyi Yi Liu added a comment - Thanks Jing Zhao for working on this. Jenkins' report has issue, TestGetContentSummaryWithSnapshot can pass locally. I also found similar issue (not the same, but also problem of getContentSummary if there is snapshot) when writing tests for large directory in HDFS-9053 , it also exists in current trunk. I apply your patch, the issue I saw is still there. I think you can fix the issue too and write test of following steps to reproduce the issue I found (Of course if you don't want to fix it, I can do it separately ): Suppose we have a directory named 'dir', create 16 files in the dir remove the last 1 file – now total 15 files in dir create a snapshot 's1' of dir add 1 file in dir – now total 16 files in dir remove the first 1 file in dir – now total 15 files in the dir call getContentSummary(dir), and then getFileCount . – the expected result is 15, but the return is 16.
        Hide
        jingzhao Jing Zhao added a comment -

        Thanks Yi! getContentSummary(dir's current path) should include (all the current files/directories) + (all the deleted files/directories but still in snapshots). Thus in the above case, the return value 16 in step 6 is correct: we have 15 files in the current dir, and the original first file in dir/.snapshot/s1.

        Show
        jingzhao Jing Zhao added a comment - Thanks Yi! getContentSummary(dir's current path) should include (all the current files/directories) + (all the deleted files/directories but still in snapshots). Thus in the above case, the return value 16 in step 6 is correct: we have 15 files in the current dir, and the original first file in dir/.snapshot/s1.
        Hide
        jingzhao Jing Zhao added a comment -

        The test failures look suspicious. I just triggered the Jenkins again.

        Show
        jingzhao Jing Zhao added a comment - The test failures look suspicious. I just triggered the Jenkins again.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 18m 4s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 0s There were no new javac warning messages.
        +1 javadoc 10m 15s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 24s The applied patch generated 2 new checkstyle issues (total was 177, now 179).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 28s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 10s Pre-build of native portion
        -1 hdfs tests 164m 2s Tests failed in hadoop-hdfs.
            209m 56s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          hadoop.hdfs.server.namenode.TestNameNodeResourceChecker



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12756374/HDFS-9063.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 58d1a02
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12511/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12511/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12511/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12511/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 4s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 0s There were no new javac warning messages. +1 javadoc 10m 15s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 24s The applied patch generated 2 new checkstyle issues (total was 177, now 179). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 10s Pre-build of native portion -1 hdfs tests 164m 2s Tests failed in hadoop-hdfs.     209m 56s   Reason Tests Failed unit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.namenode.TestNameNodeResourceChecker Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12756374/HDFS-9063.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 58d1a02 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12511/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12511/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12511/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12511/console This message was automatically generated.
        Hide
        hitliuyi Yi Liu added a comment -

        Thanks Jing for the explanation, I think I missed the fact that getContentSummary also counts the files in snapshot.

        Show
        hitliuyi Yi Liu added a comment - Thanks Jing for the explanation, I think I missed the fact that getContentSummary also counts the files in snapshot.
        Hide
        hitliuyi Yi Liu added a comment -

        +1, thanks Jing.
        I check the patch, now if getContentSummary on snapshot path, only content in snapshot is counted.

        Show
        hitliuyi Yi Liu added a comment - +1, thanks Jing. I check the patch, now if getContentSummary on snapshot path, only content in snapshot is counted.
        Hide
        hitliuyi Yi Liu added a comment -

        BTW, you can cleanup the checkstyle while committing if no other people have comments. Thanks.

        Show
        hitliuyi Yi Liu added a comment - BTW, you can cleanup the checkstyle while committing if no other people have comments. Thanks.
        Hide
        jingzhao Jing Zhao added a comment -

        Thanks for the review, Yi! I will commit the patch shortly (and also fix the 2 checkstyle warnings).

        Show
        jingzhao Jing Zhao added a comment - Thanks for the review, Yi! I will commit the patch shortly (and also fix the 2 checkstyle warnings).
        Hide
        jingzhao Jing Zhao added a comment -

        I've committed the patch into trunk and branch-2. Thanks Yi Liu for the review!

        Show
        jingzhao Jing Zhao added a comment - I've committed the patch into trunk and branch-2. Thanks Yi Liu for the review!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8481 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8481/)
        HDFS-9063. Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8481 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8481/ ) HDFS-9063 . Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #417 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/417/)
        HDFS-9063. Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #417 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/417/ ) HDFS-9063 . Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #410 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/410/)
        HDFS-9063. Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #410 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/410/ ) HDFS-9063 . Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2357 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2357/)
        HDFS-9063. Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2357 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2357/ ) HDFS-9063 . Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #1151 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1151/)
        HDFS-9063. Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1151 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1151/ ) HDFS-9063 . Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2330 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2330/)
        HDFS-9063. Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2330 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2330/ ) HDFS-9063 . Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #392 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/392/)
        HDFS-9063. Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #392 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/392/ ) HDFS-9063 . Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. (jing9: rev 3f4275310203de4ccfb15337f3c503e25408a265) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestGetContentSummaryWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
        Hide
        manojg Manoj Govindassamy added a comment -

        Hi Jing Zhao,

        I am seeing a difference in behavior on the Count command before and after this fix. Count command used to account for .snapshot directory, but after this fix, numDirs from Count command is less than expected as it is not including the .snapshot directory. Is the change in behavior intended ? Please clarify.

        Show
        manojg Manoj Govindassamy added a comment - Hi Jing Zhao , I am seeing a difference in behavior on the Count command before and after this fix. Count command used to account for .snapshot directory, but after this fix, numDirs from Count command is less than expected as it is not including the .snapshot directory. Is the change in behavior intended ? Please clarify.
        Hide
        jingzhao Jing Zhao added a comment - - edited

        Manoj Govindassamy, the patch fixed a bug and did not change the semantic of du command. Before the patch, files that were created outside of a snapshot could be included when calculating the count/usage of the snapshot. The patch fixed this bug and there is such an example in the description.

        For your case, maybe HDFS-8986 is the one related? Which Hadoop version are you using?

        Show
        jingzhao Jing Zhao added a comment - - edited Manoj Govindassamy , the patch fixed a bug and did not change the semantic of du command. Before the patch, files that were created outside of a snapshot could be included when calculating the count/usage of the snapshot. The patch fixed this bug and there is such an example in the description. For your case, maybe HDFS-8986 is the one related? Which Hadoop version are you using?
        Hide
        manojg Manoj Govindassamy added a comment -

        Jing Zhao,

        Attached a test case which shows the change in behavior after HDFS-9063 fix.

        Test does the following:
        1. Count dirs, files in a directory /foo
        2. Allow and take snapshot of the directory /foo
        3. Count dirs, files of the same directory /foo for which snapshot was taken
        4. Verify the new_dir_count == 1 + old_dir_count. Because, it should now include .snapshot directory. This verification fails after HDFS-9063 fix.

        Let me know your thoughts on the test and the expected behavior.

        Show
        manojg Manoj Govindassamy added a comment - Jing Zhao , Attached a test case which shows the change in behavior after HDFS-9063 fix. Test does the following: 1. Count dirs, files in a directory /foo 2. Allow and take snapshot of the directory /foo 3. Count dirs, files of the same directory /foo for which snapshot was taken 4. Verify the new_dir_count == 1 + old_dir_count. Because, it should now include .snapshot directory. This verification fails after HDFS-9063 fix. Let me know your thoughts on the test and the expected behavior.
        Hide
        manojg Manoj Govindassamy added a comment -

        Attached test fails on the latest trunk 3.0.0.alpha2. I didn't see the test passing after reverting HDFS-8986 fix. But test passes after reverting HDFS-9063 fix. Let me know if my understanding is wrong.

        Show
        manojg Manoj Govindassamy added a comment - Attached test fails on the latest trunk 3.0.0.alpha2. I didn't see the test passing after reverting HDFS-8986 fix. But test passes after reverting HDFS-9063 fix. Let me know if my understanding is wrong.
        Hide
        jingzhao Jing Zhao added a comment -

        Ah, I see what you mean. Yes, we have a minor incompatibility here, please see my first comment:

        In the meanwhile, the snapshot diff list size is no longer included in the computation result. This can bring minor incompatibility but is consistent with the change in HDFS-7728.

        I.e., since we already have a number in the content summary to indicate the total number of snapshots, the number of snapshots is no longer added into the directory number. To me the old behavior is more like a bug, but we can still mark this jira as incompatible maybe.

        Show
        jingzhao Jing Zhao added a comment - Ah, I see what you mean. Yes, we have a minor incompatibility here, please see my first comment: In the meanwhile, the snapshot diff list size is no longer included in the computation result. This can bring minor incompatibility but is consistent with the change in HDFS-7728 . I.e., since we already have a number in the content summary to indicate the total number of snapshots, the number of snapshots is no longer added into the directory number. To me the old behavior is more like a bug, but we can still mark this jira as incompatible maybe.
        Hide
        manojg Manoj Govindassamy added a comment -


        Thanks for looking into this. Thats right, ContentSummary does have good info about Snapshots and I do feel its better to keep dirCount and snapCount separate. However, the behavior of the Count command has changed and it would be good to mark this jira as incompatible.

        Show
        manojg Manoj Govindassamy added a comment - Thanks for looking into this. Thats right, ContentSummary does have good info about Snapshots and I do feel its better to keep dirCount and snapCount separate. However, the behavior of the Count command has changed and it would be good to mark this jira as incompatible.
        Hide
        jingzhao Jing Zhao added a comment -

        Agree. Added the incompatible label and also updated the release note. Thanks for looking into this, Manoj Govindassamy.

        Show
        jingzhao Jing Zhao added a comment - Agree. Added the incompatible label and also updated the release note. Thanks for looking into this, Manoj Govindassamy .
        Hide
        manojg Manoj Govindassamy added a comment -

        Jing Zhao,

        Tried querying ContentSummary for the Directory which has snapshots, and still I don't see ContentSummary's SnpashotDirectoryCount reflecting any snapshots taken. Is there any other way of querying the ContentSummary which will reflect the right Snap Dir Count ? Please clarify.

        Output from attached test v002.

        ContentSummary: /foo - Dir: 2, SnapDir: 0, SnapFile: 0
        ContentSummary: /foo/bar - Dir: 1, SnapDir: 0, SnapFile: 0
        Count /foo -         none             inf            none             inf            2            0                  0 /foo
        Created Snaphot: /foo/.snapshot/s1
        Created Snaphot: /foo/.snapshot/s2
        Count /foo -         none             inf            none             inf            2            0                  0 /foo
        ContentSummary: /foo - Dir: 2, SnapDir: 0, SnapFile: 0
        ContentSummary: /foo/bar - Dir: 1, SnapDir: 0, SnapFile: 0
        ContentSummary: /foo/.snapshot/s1 - Dir: 2, SnapDir: 0, SnapFile: 0
        ContentSummary: /foo/.snapshot/s2 - Dir: 2, SnapDir: 0, SnapFile: 0
        
        Show
        manojg Manoj Govindassamy added a comment - Jing Zhao , Tried querying ContentSummary for the Directory which has snapshots, and still I don't see ContentSummary's SnpashotDirectoryCount reflecting any snapshots taken. Is there any other way of querying the ContentSummary which will reflect the right Snap Dir Count ? Please clarify. Output from attached test v002. ContentSummary: /foo - Dir: 2, SnapDir: 0, SnapFile: 0 ContentSummary: /foo/bar - Dir: 1, SnapDir: 0, SnapFile: 0 Count /foo - none inf none inf 2 0 0 /foo Created Snaphot: /foo/.snapshot/s1 Created Snaphot: /foo/.snapshot/s2 Count /foo - none inf none inf 2 0 0 /foo ContentSummary: /foo - Dir: 2, SnapDir: 0, SnapFile: 0 ContentSummary: /foo/bar - Dir: 1, SnapDir: 0, SnapFile: 0 ContentSummary: /foo/.snapshot/s1 - Dir: 2, SnapDir: 0, SnapFile: 0 ContentSummary: /foo/.snapshot/s2 - Dir: 2, SnapDir: 0, SnapFile: 0
        Hide
        manojg Manoj Govindassamy added a comment -

        Jing Zhao,

        Did you get a chance to look at the attached test case ? Is there anything wrong with the expectation in the test ? I can file a jira to track this issue, if you also feel snapDir count is not working as expected. Thanks.

        Show
        manojg Manoj Govindassamy added a comment - Jing Zhao , Did you get a chance to look at the attached test case ? Is there anything wrong with the expectation in the test ? I can file a jira to track this issue, if you also feel snapDir count is not working as expected. Thanks.
        Hide
        xiaochen Xiao Chen added a comment -

        Hm, I'm a little confused by this comment:

        I.e., since we already have a number in the content summary to indicate the total number of snapshots, the number of snapshots is no longer added into the directory number.

        Jing Zhao, could you clarify?

        I have just chatted offline with Manoj. The snapshotXXXX in ContentSummary, added by HDFS-8986, was to track how many files/directories are created under a snapshot, so that count -x could exclude them and calculate a result of direct usage. It is not tracking how many snapshots have been taken on a directory.

        Do we have number of snapshots visible to users? I only found SnapshotManager#snapshotCounter at the hdfs level, and DirectorySnapshottableFeature#getNumSnapshots at the dir level - neither seems to be propagating to any shell commands.

        My 2 cents are this is will not reasonably impact quotas, so maybe good as-is. Nice to have this displayed too, if users want to get it absolutely accurate. Please share your thoughts, thanks!

        Show
        xiaochen Xiao Chen added a comment - Hm, I'm a little confused by this comment: I.e., since we already have a number in the content summary to indicate the total number of snapshots, the number of snapshots is no longer added into the directory number. Jing Zhao , could you clarify? I have just chatted offline with Manoj. The snapshotXXXX in ContentSummary , added by HDFS-8986 , was to track how many files/directories are created under a snapshot, so that count -x could exclude them and calculate a result of direct usage. It is not tracking how many snapshots have been taken on a directory. Do we have number of snapshots visible to users? I only found SnapshotManager#snapshotCounter at the hdfs level, and DirectorySnapshottableFeature#getNumSnapshots at the dir level - neither seems to be propagating to any shell commands. My 2 cents are this is will not reasonably impact quotas, so maybe good as-is. Nice to have this displayed too, if users want to get it absolutely accurate. Please share your thoughts, thanks!
        Hide
        manojg Manoj Govindassamy added a comment -

        Thanks for the clarification Xiao Chen. I was mistaking SnpashotDirectoryCount to include snapshot count also. After looking at the code, I see the raw snapshot count is not exposed in ContentSummary or Count command.

        Show
        manojg Manoj Govindassamy added a comment - Thanks for the clarification Xiao Chen . I was mistaking SnpashotDirectoryCount to include snapshot count also. After looking at the code, I see the raw snapshot count is not exposed in ContentSummary or Count command.

          People

          • Assignee:
            jingzhao Jing Zhao
            Reporter:
            jingzhao Jing Zhao
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development