Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8986

Add option to -du to calculate directory space usage excluding snapshots

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Add a -x option for "hdfs -du" and "hdfs -count" commands to exclude snapshots from being calculated.

      Description

      When running hadoop fs -du on a snapshotted directory (or one of its children), the report includes space consumed by blocks that are only present in the snapshots. This is confusing for end users.

      $  hadoop fs -du -h -s /tmp/parent /tmp/parent/*
      799.7 M  2.3 G  /tmp/parent
      799.7 M  2.3 G  /tmp/parent/sub1
      
      $ hdfs dfs -createSnapshot /tmp/parent snap1
      Created snapshot /tmp/parent/.snapshot/snap1
      
      $ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
      ...
      
      $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
      799.7 M  2.3 G  /tmp/parent
      799.7 M  2.3 G  /tmp/parent/sub1
      
      $ hdfs dfs -deleteSnapshot /tmp/parent snap1
      $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
      0  0  /tmp/parent
      0  0  /tmp/parent/sub1
      

      It would be helpful if we had a flag, say -X, to exclude any snapshot related disk usage in the output

      1. HDFS-8986.01.patch
        34 kB
        Xiao Chen
      2. HDFS-8986.02.patch
        38 kB
        Xiao Chen
      3. HDFS-8986.03.patch
        39 kB
        Xiao Chen
      4. HDFS-8986.04.patch
        39 kB
        Xiao Chen
      5. HDFS-8986.05.patch
        39 kB
        Xiao Chen
      6. HDFS-8986.06.patch
        39 kB
        Wei-Chiu Chuang
      7. HDFS-8986.07.patch
        40 kB
        Xiao Chen
      8. HDFS-8986.branch-2.patch
        39 kB
        Xiao Chen

        Issue Links

          Activity

          Hide
          jagadesh.kiran Jagadesh Kiran N added a comment -

          Adding -X exclude for the disk usage may not be the correct option. If the concern is only for the confusion for the user, IMO we can mention a note about this in the command usage guide.If the directory is snapshottable, diskusage may be different from expected value because some deletions may not remove content from the disk

          Show
          jagadesh.kiran Jagadesh Kiran N added a comment - Adding -X exclude for the disk usage may not be the correct option. If the concern is only for the confusion for the user, IMO we can mention a note about this in the command usage guide.If the directory is snapshottable, diskusage may be different from expected value because some deletions may not remove content from the disk
          Hide
          ggop Gautam Gopalakrishnan added a comment -

          Thanks Jagadesh Kiran N. This is a real need for many users and something that works until snapshots are enabled. Our tools should continue to work in a familiar way in the presence of new features or have a way of compensating for it. Other than documenting this change, are you aware of any other method to achieve what the users want (get current usage of a directory)?

          Show
          ggop Gautam Gopalakrishnan added a comment - Thanks Jagadesh Kiran N . This is a real need for many users and something that works until snapshots are enabled. Our tools should continue to work in a familiar way in the presence of new features or have a way of compensating for it. Other than documenting this change, are you aware of any other method to achieve what the users want (get current usage of a directory)?
          Hide
          jagadesh.kiran Jagadesh Kiran N added a comment -

          Hi Gautam Gopalakrishnan ,IMO snapshots are not treated as normal directories, it will have admin privileges . Hence no need to mention normal user that this is snapshot directory. Please let me know if you have any specific usecase for the same or Please share any inputs if you have already thought of.

          Show
          jagadesh.kiran Jagadesh Kiran N added a comment - Hi Gautam Gopalakrishnan ,IMO snapshots are not treated as normal directories, it will have admin privileges . Hence no need to mention normal user that this is snapshot directory. Please let me know if you have any specific usecase for the same or Please share any inputs if you have already thought of.
          Hide
          ggop Gautam Gopalakrishnan added a comment -

          Jagadesh Kiran N I was not referring to the .snapshot directory itself. Admins can turn snapshots on and off on a directory (or a parent) and a normal user's use of the -du command will vary depending on how many snapshots hold older blocks. To an end user the use of snapshots should be transparent, in this case it is not.

          Let's say home directories are under /user and you as the hdfs admin enable snapshots on /user. Jane Doe could figure out how much space she uses by running a -du on /user/jdoe. Now with snapshots enabled, this measurement is no longer useful. End users shouldn't care that snapshots are on or off, the -du command should work as it always has. Alternatively, users should have a tool that allows them to measure usage within a directory excluding snapshots. Does this help?

          Show
          ggop Gautam Gopalakrishnan added a comment - Jagadesh Kiran N I was not referring to the .snapshot directory itself. Admins can turn snapshots on and off on a directory (or a parent) and a normal user's use of the -du command will vary depending on how many snapshots hold older blocks. To an end user the use of snapshots should be transparent, in this case it is not. Let's say home directories are under /user and you as the hdfs admin enable snapshots on /user . Jane Doe could figure out how much space she uses by running a -du on /user/jdoe . Now with snapshots enabled, this measurement is no longer useful. End users shouldn't care that snapshots are on or off, the -du command should work as it always has. Alternatively, users should have a tool that allows them to measure usage within a directory excluding snapshots. Does this help?
          Hide
          jagadesh.kiran Jagadesh Kiran N added a comment -

          Thanks for your cliarification Gautam Gopalakrishnan . Tsz Wo Nicholas Sze & Jing Zhao Please share your views about this improvement

          Show
          jagadesh.kiran Jagadesh Kiran N added a comment - Thanks for your cliarification Gautam Gopalakrishnan . Tsz Wo Nicholas Sze & Jing Zhao Please share your views about this improvement
          Hide
          qwertymaniac Harsh J added a comment -

          Jagadesh Kiran N - What's the precise opposition w.r.t. adding a new flag to explicitly exclude snapshot counts? It does not break compatibility, and is an added feature/improvement.

          Show
          qwertymaniac Harsh J added a comment - Jagadesh Kiran N - What's the precise opposition w.r.t. adding a new flag to explicitly exclude snapshot counts? It does not break compatibility, and is an added feature/improvement.
          Hide
          jagadesh.kiran Jagadesh Kiran N added a comment -

          Harsh J i am not opposing the same , i was collecting the opinions before i start implementing the same.

          Show
          jagadesh.kiran Jagadesh Kiran N added a comment - Harsh J i am not opposing the same , i was collecting the opinions before i start implementing the same.
          Hide
          qwertymaniac Harsh J added a comment -

          This continues to cause a bunch of confusion among our user-base who are still reliant on the pre-snapshot feature behaviour, so it would be nice to see it implemented.

          Show
          qwertymaniac Harsh J added a comment - This continues to cause a bunch of confusion among our user-base who are still reliant on the pre-snapshot feature behaviour, so it would be nice to see it implemented.
          Hide
          jagadesh.kiran Jagadesh Kiran N added a comment -

          Thanks Harsh J , i am working on this will try to provide patch next week, busy with other work as of now

          Show
          jagadesh.kiran Jagadesh Kiran N added a comment - Thanks Harsh J , i am working on this will try to provide patch next week, busy with other work as of now
          Hide
          cnauroth Chris Nauroth added a comment -

          I see -du mentioned here. I recommend that we support the same functionality for -count too.

          Show
          cnauroth Chris Nauroth added a comment - I see -du mentioned here. I recommend that we support the same functionality for -count too.
          Hide
          cnauroth Chris Nauroth added a comment -

          Within the scope of this JIRA, I also suggest that we update the documentation (hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsSnapshots.md) to give a thorough description of the behavior of -du and -count in the presence of snapshots. The current behavior is helpful for seeing the full metadata cost of a path, but it can be a source of confusion for users.

          Show
          cnauroth Chris Nauroth added a comment - Within the scope of this JIRA, I also suggest that we update the documentation (hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsSnapshots.md) to give a thorough description of the behavior of -du and -count in the presence of snapshots. The current behavior is helpful for seeing the full metadata cost of a path, but it can be a source of confusion for users.
          Hide
          ggop Gautam Gopalakrishnan added a comment -

          I feel I should invert the request in this jira. The behaviour of -du should be the same regardless of whether snapshots are present or not. We should add a flag to include snapshot volume, rather than exclude. Possibly the same change is required for -count. Chris Nauroth and Harsh J, what do you think?

          Jagadesh Kiran N I can take this jira if you're busy with other work.

          Show
          ggop Gautam Gopalakrishnan added a comment - I feel I should invert the request in this jira. The behaviour of -du should be the same regardless of whether snapshots are present or not. We should add a flag to include snapshot volume, rather than exclude. Possibly the same change is required for -count . Chris Nauroth and Harsh J , what do you think? Jagadesh Kiran N I can take this jira if you're busy with other work.
          Hide
          cnauroth Chris Nauroth added a comment -

          I don't think we can change the default behavior of the commands, at least not within 2.x, on grounds of backwards-compatibility. It's possible that users already depend on inclusion of snapshot contents in the results of these commands. Adding a new option for filtering out snapshot contents would be a backwards-compatible change though.

          Show
          cnauroth Chris Nauroth added a comment - I don't think we can change the default behavior of the commands, at least not within 2.x, on grounds of backwards-compatibility. It's possible that users already depend on inclusion of snapshot contents in the results of these commands. Adding a new option for filtering out snapshot contents would be a backwards-compatible change though.
          Hide
          jagadesh.kiran Jagadesh Kiran N added a comment -

          no issues Gautam Gopalakrishnan ,you can assign

          Show
          jagadesh.kiran Jagadesh Kiran N added a comment - no issues Gautam Gopalakrishnan ,you can assign
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Gautam for creating this, and everyone for the discussion.

          I feel adding an option to exclude would be the better option to go for compatibility reasons. We should also update the doc to include this, and also document the current behavior of the going 1-level deep logic (code here).

          Gautam Gopalakrishnan, are you working on this? I can work on it or continue on what you already have, if you're busy.

          Show
          xiaochen Xiao Chen added a comment - Thanks Gautam for creating this, and everyone for the discussion. I feel adding an option to exclude would be the better option to go for compatibility reasons. We should also update the doc to include this, and also document the current behavior of the going 1-level deep logic (code here ). Gautam Gopalakrishnan , are you working on this? I can work on it or continue on what you already have, if you're busy.
          Hide
          ggop Gautam Gopalakrishnan added a comment -

          Xiao Chen Thanks for the offer, please take ownership when you're able to. I am unable to work on this at present.

          Show
          ggop Gautam Gopalakrishnan added a comment - Xiao Chen Thanks for the offer, please take ownership when you're able to. I am unable to work on this at present.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Gautam for the response. I'm assigning this to myself.

          Show
          xiaochen Xiao Chen added a comment - Thanks Gautam for the response. I'm assigning this to myself.
          Hide
          xiaochen Xiao Chen added a comment -

          I'm attaching a preliminary patch 1 to show the general idea of this change.

          I think Chris' comment of modifying -count together with -du makes perfect sense, since they are the 2 usages of FileSystem#getContentSummary in the shell. (Except for -rm's internal usage, which is not in the scope of this exclude snapshot discussion.)

          The general idea is to add a -x flag to the shell commands, to exclude snapshots from calculation.

          Implementation-wise, since getContentSummary is a public API on FileSystem, I don't think we can change anything there. Alternatively, I added fields and methods to ContentSummary (which is InterfaceStability.Evolving - and I think the changes here doesn't break compatibility anyway), to store snapshot-related values in the same object. Then at the caller of getContentSummary, one can subtract the snapshot-related values from the total, to get the desired -x result.
          The calculation is done by adding a snapshot specific ContentCounts to the ContentSummaryComputationContext object.

          Please review and provide feedback regarding this approach. I plan to polish the tests/docs in a later rev. Thanks very much!

          Show
          xiaochen Xiao Chen added a comment - I'm attaching a preliminary patch 1 to show the general idea of this change. I think Chris' comment of modifying -count together with -du makes perfect sense, since they are the 2 usages of FileSystem#getContentSummary in the shell. (Except for -rm 's internal usage, which is not in the scope of this exclude snapshot discussion.) The general idea is to add a -x flag to the shell commands, to exclude snapshots from calculation. Implementation-wise, since getContentSummary is a public API on FileSystem , I don't think we can change anything there. Alternatively, I added fields and methods to ContentSummary (which is InterfaceStability.Evolving - and I think the changes here doesn't break compatibility anyway), to store snapshot-related values in the same object. Then at the caller of getContentSummary , one can subtract the snapshot-related values from the total, to get the desired -x result. The calculation is done by adding a snapshot specific ContentCounts to the ContentSummaryComputationContext object. Please review and provide feedback regarding this approach. I plan to polish the tests/docs in a later rev. Thanks very much!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 6m 55s trunk passed
          +1 compile 5m 57s trunk passed with JDK v1.8.0_77
          +1 compile 6m 52s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 6s trunk passed
          +1 mvnsite 2m 26s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 5m 3s trunk passed
          +1 javadoc 2m 23s trunk passed with JDK v1.8.0_77
          +1 javadoc 3m 15s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 59s the patch passed
          +1 compile 5m 49s the patch passed with JDK v1.8.0_77
          +1 cc 5m 49s the patch passed
          +1 javac 5m 49s the patch passed
          +1 compile 6m 45s the patch passed with JDK v1.7.0_95
          +1 cc 6m 45s the patch passed
          +1 javac 6m 45s the patch passed
          -1 checkstyle 1m 6s root: patch generated 9 new + 177 unchanged - 0 fixed = 186 total (was 177)
          +1 mvnsite 2m 19s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 5m 53s the patch passed
          -1 javadoc 4m 55s hadoop-common-project_hadoop-common-jdk1.8.0_77 with JDK v1.8.0_77 generated 12 new + 1 unchanged - 0 fixed = 13 total (was 1)
          +1 javadoc 2m 23s the patch passed with JDK v1.8.0_77
          -1 javadoc 8m 39s hadoop-common-project_hadoop-common-jdk1.7.0_95 with JDK v1.7.0_95 generated 12 new + 13 unchanged - 0 fixed = 25 total (was 13)
          +1 javadoc 3m 23s the patch passed with JDK v1.7.0_95
          -1 unit 7m 24s hadoop-common in the patch failed with JDK v1.8.0_77.
          +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77.
          -1 unit 68m 58s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
          -1 unit 7m 16s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 67m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          221m 3s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.fs.shell.TestCount
            hadoop.cli.TestCLI
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
          JDK v1.7.0_95 Failed junit tests hadoop.fs.shell.TestCount
            hadoop.cli.TestCLI
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.TestDFSShell
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798663/HDFS-8986.01.patch
          JIRA Issue HDFS-8986
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux df3b2fd4b254 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 69f3d42
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/diff-checkstyle-root.txt
          javadoc hadoop-common-project_hadoop-common-jdk1.8.0_77: https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt
          javadoc hadoop-common-project_hadoop-common-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15175/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15175/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 6m 55s trunk passed +1 compile 5m 57s trunk passed with JDK v1.8.0_77 +1 compile 6m 52s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 6s trunk passed +1 mvnsite 2m 26s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 5m 3s trunk passed +1 javadoc 2m 23s trunk passed with JDK v1.8.0_77 +1 javadoc 3m 15s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 59s the patch passed +1 compile 5m 49s the patch passed with JDK v1.8.0_77 +1 cc 5m 49s the patch passed +1 javac 5m 49s the patch passed +1 compile 6m 45s the patch passed with JDK v1.7.0_95 +1 cc 6m 45s the patch passed +1 javac 6m 45s the patch passed -1 checkstyle 1m 6s root: patch generated 9 new + 177 unchanged - 0 fixed = 186 total (was 177) +1 mvnsite 2m 19s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 53s the patch passed -1 javadoc 4m 55s hadoop-common-project_hadoop-common-jdk1.8.0_77 with JDK v1.8.0_77 generated 12 new + 1 unchanged - 0 fixed = 13 total (was 1) +1 javadoc 2m 23s the patch passed with JDK v1.8.0_77 -1 javadoc 8m 39s hadoop-common-project_hadoop-common-jdk1.7.0_95 with JDK v1.7.0_95 generated 12 new + 13 unchanged - 0 fixed = 25 total (was 13) +1 javadoc 3m 23s the patch passed with JDK v1.7.0_95 -1 unit 7m 24s hadoop-common in the patch failed with JDK v1.8.0_77. +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77. -1 unit 68m 58s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 7m 16s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 67m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 221m 3s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.fs.shell.TestCount   hadoop.cli.TestCLI   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.shortcircuit.TestShortCircuitCache JDK v1.7.0_95 Failed junit tests hadoop.fs.shell.TestCount   hadoop.cli.TestCLI   hadoop.hdfs.TestHFlush   hadoop.hdfs.TestDFSShell   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798663/HDFS-8986.01.patch JIRA Issue HDFS-8986 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux df3b2fd4b254 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 69f3d42 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/diff-checkstyle-root.txt javadoc hadoop-common-project_hadoop-common-jdk1.8.0_77: https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt javadoc hadoop-common-project_hadoop-common-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15175/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15175/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15175/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 2 fixes all reported errors. The 2 added tests in DFSShell can pass locally, not sure why it failed on jenkins.

          Show
          xiaochen Xiao Chen added a comment - Patch 2 fixes all reported errors. The 2 added tests in DFSShell can pass locally, not sure why it failed on jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 32s trunk passed
          +1 compile 5m 42s trunk passed with JDK v1.8.0_77
          +1 compile 6m 37s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 6s trunk passed
          +1 mvnsite 2m 20s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 5m 0s trunk passed
          +1 javadoc 2m 16s trunk passed with JDK v1.8.0_77
          +1 javadoc 3m 13s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 54s the patch passed
          +1 compile 5m 52s the patch passed with JDK v1.8.0_77
          +1 cc 5m 52s the patch passed
          +1 javac 5m 52s the patch passed
          +1 compile 6m 39s the patch passed with JDK v1.7.0_95
          +1 cc 6m 39s the patch passed
          +1 javac 6m 39s the patch passed
          -1 checkstyle 1m 4s root: patch generated 3 new + 166 unchanged - 11 fixed = 169 total (was 177)
          +1 mvnsite 2m 20s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 5m 43s the patch passed
          +1 javadoc 2m 19s the patch passed with JDK v1.8.0_77
          +1 javadoc 3m 12s the patch passed with JDK v1.7.0_95
          -1 unit 16m 51s hadoop-common in the patch failed with JDK v1.8.0_77.
          +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77.
          -1 unit 58m 40s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
          +1 unit 8m 23s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 55m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          206m 37s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.TestFileAppend
          JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.TestHFlush



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799339/HDFS-8986.02.patch
          JIRA Issue HDFS-8986
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml cc
          uname Linux 22eef5049230 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4770037
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15189/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15189/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15189/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15189/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15189/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15189/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15189/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15189/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15189/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 32s trunk passed +1 compile 5m 42s trunk passed with JDK v1.8.0_77 +1 compile 6m 37s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 6s trunk passed +1 mvnsite 2m 20s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 5m 0s trunk passed +1 javadoc 2m 16s trunk passed with JDK v1.8.0_77 +1 javadoc 3m 13s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 54s the patch passed +1 compile 5m 52s the patch passed with JDK v1.8.0_77 +1 cc 5m 52s the patch passed +1 javac 5m 52s the patch passed +1 compile 6m 39s the patch passed with JDK v1.7.0_95 +1 cc 6m 39s the patch passed +1 javac 6m 39s the patch passed -1 checkstyle 1m 4s root: patch generated 3 new + 166 unchanged - 11 fixed = 169 total (was 177) +1 mvnsite 2m 20s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 5m 43s the patch passed +1 javadoc 2m 19s the patch passed with JDK v1.8.0_77 +1 javadoc 3m 12s the patch passed with JDK v1.7.0_95 -1 unit 16m 51s hadoop-common in the patch failed with JDK v1.8.0_77. +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_77. -1 unit 58m 40s hadoop-hdfs in the patch failed with JDK v1.8.0_77. +1 unit 8m 23s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 55m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 206m 37s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.TestFileAppend JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_95 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.TestHFlush Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799339/HDFS-8986.02.patch JIRA Issue HDFS-8986 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml cc uname Linux 22eef5049230 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4770037 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15189/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15189/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15189/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15189/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15189/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15189/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15189/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15189/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15189/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          The left test failures are not related (HDFS-10291 fixes TestShortCircuitLocalRead). The checkstyle I think should be left as such to make the code more readable.

          Appreciate any review comments. Thanks!

          Show
          xiaochen Xiao Chen added a comment - The left test failures are not related ( HDFS-10291 fixes TestShortCircuitLocalRead). The checkstyle I think should be left as such to make the code more readable. Appreciate any review comments. Thanks!
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Hi Xiao Chen thanks for working on the patch! I'd like to push this forward.

          • in ContentSummary.java, the name of setter method for snapshotLength, snapshotFileCount, snapshotDirectoryCount and snapshotSpaceConsumed should be prefixed by "set". E.g. setSnapshotLength
          • in ContentSummary#equals(), you may declare a ContentSummary object and typecast the to object to it, so as to avoid explicitly typecasting every method call. This is just a personal taste, not big deal though.
          • Please update FileSystemShell.md to include the -x option for the usage of du.
            Usage: `hadoop fs -du [-s] [-h] URI [URI ...]`
          • I don't understand this code in INodeDirectory, and I wonder if it has a bug. If I understand it correctly, the counts field and snapshotCounts field of summary object will be exactly the same. On the contrary, I think you may have to declare another method similar to DirectoryWithSnapshotFeature.computeContentSummary4Snapshot, but which computes content for snapshottable subdirectories and files only.

            // if the getContentSummary call is against a non-snapshot path, the
            // computation should include all the deleted files/directories
            sf.computeContentSummary4Snapshot(summary.getBlockStoragePolicySuite(),
            summary.getCounts());
            // Also compute ContentSummary for snapshotCounts
            sf.computeContentSummary4Snapshot(summary.getBlockStoragePolicySuite(),
            summary.getSnapshotCounts());

          Show
          jojochuang Wei-Chiu Chuang added a comment - Hi Xiao Chen thanks for working on the patch! I'd like to push this forward. in ContentSummary.java, the name of setter method for snapshotLength , snapshotFileCount , snapshotDirectoryCount and snapshotSpaceConsumed should be prefixed by "set". E.g. setSnapshotLength in ContentSummary#equals() , you may declare a ContentSummary object and typecast the to object to it, so as to avoid explicitly typecasting every method call. This is just a personal taste, not big deal though. Please update FileSystemShell.md to include the -x option for the usage of du. Usage: `hadoop fs -du [-s] [-h] URI [URI ...]` I don't understand this code in INodeDirectory, and I wonder if it has a bug. If I understand it correctly, the counts field and snapshotCounts field of summary object will be exactly the same. On the contrary, I think you may have to declare another method similar to DirectoryWithSnapshotFeature.computeContentSummary4Snapshot , but which computes content for snapshottable subdirectories and files only. // if the getContentSummary call is against a non-snapshot path, the // computation should include all the deleted files/directories sf.computeContentSummary4Snapshot(summary.getBlockStoragePolicySuite(), summary.getCounts()); // Also compute ContentSummary for snapshotCounts sf.computeContentSummary4Snapshot(summary.getBlockStoragePolicySuite(), summary.getSnapshotCounts());
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 33s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 38m 57s Maven dependency ordering for branch
          +1 mvninstall 9m 5s trunk passed
          +1 compile 7m 56s trunk passed
          +1 checkstyle 1m 37s trunk passed
          +1 mvnsite 2m 36s trunk passed
          +1 mvneclipse 0m 44s trunk passed
          +1 findbugs 4m 38s trunk passed
          +1 javadoc 2m 16s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 2m 0s the patch passed
          +1 compile 7m 1s the patch passed
          +1 cc 7m 1s the patch passed
          +1 javac 7m 1s the patch passed
          -0 checkstyle 1m 38s root: The patch generated 11 new + 351 unchanged - 12 fixed = 362 total (was 363)
          +1 mvnsite 2m 33s the patch passed
          +1 mvneclipse 0m 44s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 5m 12s the patch passed
          +1 javadoc 2m 16s the patch passed
          +1 unit 9m 11s hadoop-common in the patch passed.
          +1 unit 1m 9s hadoop-hdfs-client in the patch passed.
          +1 unit 77m 2s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          178m 50s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799339/HDFS-8986.02.patch
          JIRA Issue HDFS-8986
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml cc
          uname Linux 24005fc823e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 132deb4
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16154/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16154/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16154/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 33s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 38m 57s Maven dependency ordering for branch +1 mvninstall 9m 5s trunk passed +1 compile 7m 56s trunk passed +1 checkstyle 1m 37s trunk passed +1 mvnsite 2m 36s trunk passed +1 mvneclipse 0m 44s trunk passed +1 findbugs 4m 38s trunk passed +1 javadoc 2m 16s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 0s the patch passed +1 compile 7m 1s the patch passed +1 cc 7m 1s the patch passed +1 javac 7m 1s the patch passed -0 checkstyle 1m 38s root: The patch generated 11 new + 351 unchanged - 12 fixed = 362 total (was 363) +1 mvnsite 2m 33s the patch passed +1 mvneclipse 0m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 5m 12s the patch passed +1 javadoc 2m 16s the patch passed +1 unit 9m 11s hadoop-common in the patch passed. +1 unit 1m 9s hadoop-hdfs-client in the patch passed. +1 unit 77m 2s hadoop-hdfs in the patch passed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 178m 50s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799339/HDFS-8986.02.patch JIRA Issue HDFS-8986 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml cc uname Linux 24005fc823e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 132deb4 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16154/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16154/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16154/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks a lot Wei-Chiu Chuang for the review! New patch attached with comments inline:

          in ContentSummary.java, the name of setter method for snapshotLength, snapshotFileCount, snapshotDirectoryCount and snapshotSpaceConsumed should be prefixed by "set". E.g. setSnapshotLength

          I agree setXXX is a better setter name. The reason in these names here is for consistency with existing setter method naming. It's a public (though evolving) API, so I'd want to keep the change minimal.

          in ContentSummary#equals(), you may declare a ContentSummary object and typecast the to object to it, so as to avoid explicitly typecasting every method call. This is just a personal taste, not big deal though.

          Good idea, updated.

          Please update FileSystemShell.md to include the -x option for the usage of du.

          Good catch! Updated.

          I don't understand this code in INodeDirectory, and I wonder if it has a bug. If I understand it correctly, the counts field and snapshotCounts field of summary object will be exactly the same. On the contrary, I think you may have to declare another method similar to DirectoryWithSnapshotFeature.computeContentSummary4Snapshot, but which computes content for snapshottable subdirectories and files only.

          I think current patch is correct. It's a bit difficult to read through, since (the great change) of HDFS-4995. But the high level idea is that, ContentCounts is aggregated calculation. You're right in that the calculation in INodeDirectory#computeContentSummary would aggregate same values into counts and snapshotCounts, but that's what we want. This way, in the final calculation in FsUsage$Du#processPath we can exclude the snapshot portion from the calculation by (All - snapshotAll).
          I added 1 more step in the test to create a file as well, after snapshot taken. Makes sense?

          Show
          xiaochen Xiao Chen added a comment - Thanks a lot Wei-Chiu Chuang for the review! New patch attached with comments inline: in ContentSummary.java, the name of setter method for snapshotLength, snapshotFileCount, snapshotDirectoryCount and snapshotSpaceConsumed should be prefixed by "set". E.g. setSnapshotLength I agree setXXX is a better setter name. The reason in these names here is for consistency with existing setter method naming. It's a public (though evolving) API, so I'd want to keep the change minimal. in ContentSummary#equals(), you may declare a ContentSummary object and typecast the to object to it, so as to avoid explicitly typecasting every method call. This is just a personal taste, not big deal though. Good idea, updated. Please update FileSystemShell.md to include the -x option for the usage of du. Good catch! Updated. I don't understand this code in INodeDirectory, and I wonder if it has a bug. If I understand it correctly, the counts field and snapshotCounts field of summary object will be exactly the same. On the contrary, I think you may have to declare another method similar to DirectoryWithSnapshotFeature.computeContentSummary4Snapshot, but which computes content for snapshottable subdirectories and files only. I think current patch is correct. It's a bit difficult to read through, since (the great change) of HDFS-4995 . But the high level idea is that, ContentCounts is aggregated calculation. You're right in that the calculation in INodeDirectory#computeContentSummary would aggregate same values into counts and snapshotCounts , but that's what we want. This way, in the final calculation in FsUsage$Du#processPath we can exclude the snapshot portion from the calculation by (All - snapshotAll). I added 1 more step in the test to create a file as well, after snapshot taken. Makes sense?
          Hide
          xiaochen Xiao Chen added a comment -

          Reattaching patch 3 to also add the [-x] to count's FileSystemShell doc.

          Show
          xiaochen Xiao Chen added a comment - Reattaching patch 3 to also add the [-x] to count's FileSystemShell doc.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 7m 14s trunk passed
          +1 compile 7m 7s trunk passed
          +1 checkstyle 1m 28s trunk passed
          +1 mvnsite 2m 20s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 4m 26s trunk passed
          +1 javadoc 2m 1s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 2m 8s the patch passed
          +1 compile 7m 28s the patch passed
          +1 cc 7m 28s the patch passed
          +1 javac 7m 28s the patch passed
          -0 checkstyle 1m 26s root: The patch generated 1 new + 352 unchanged - 11 fixed = 353 total (was 363)
          +1 mvnsite 2m 21s the patch passed
          +1 mvneclipse 0m 40s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 5m 58s the patch passed
          +1 javadoc 2m 6s the patch passed
          -1 unit 20m 18s hadoop-common in the patch failed.
          +1 unit 0m 58s hadoop-hdfs-client in the patch passed.
          -1 unit 76m 53s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 26s The patch does not generate ASF License warnings.
          147m 48s



          Reason Tests
          Failed junit tests hadoop.cli.TestHDFSCLI
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.namenode.ha.TestHAAppend
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820537/HDFS-8986.03.patch
          JIRA Issue HDFS-8986
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml cc
          uname Linux 627a5e33c365 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 54fe17a
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16217/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16217/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16217/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16217/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16217/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 7m 14s trunk passed +1 compile 7m 7s trunk passed +1 checkstyle 1m 28s trunk passed +1 mvnsite 2m 20s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 4m 26s trunk passed +1 javadoc 2m 1s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 8s the patch passed +1 compile 7m 28s the patch passed +1 cc 7m 28s the patch passed +1 javac 7m 28s the patch passed -0 checkstyle 1m 26s root: The patch generated 1 new + 352 unchanged - 11 fixed = 353 total (was 363) +1 mvnsite 2m 21s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 5m 58s the patch passed +1 javadoc 2m 6s the patch passed -1 unit 20m 18s hadoop-common in the patch failed. +1 unit 0m 58s hadoop-hdfs-client in the patch passed. -1 unit 76m 53s hadoop-hdfs in the patch failed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 147m 48s Reason Tests Failed junit tests hadoop.cli.TestHDFSCLI   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.namenode.ha.TestHAAppend Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820537/HDFS-8986.03.patch JIRA Issue HDFS-8986 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml cc uname Linux 627a5e33c365 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 54fe17a Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16217/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16217/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16217/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16217/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16217/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          I had a round of self-review. Patch 4 is attached with some comments in triky areas, and a checkstyle fix.

          I should also mention that, with this approach, there's more overhead. This is true even when -x is not given - the behavior is compatible still though.

          • calculation for snapshots are always performed.
          • ContentSummary is bigger than before (since it holds more long fields) so we need to transfer a larger object.

          IMHO, since HDFS-4995 adds more control over the calculation, and we're not creating a lot of ContentSummary objects, the above 2 things should be fine.

          Show
          xiaochen Xiao Chen added a comment - I had a round of self-review. Patch 4 is attached with some comments in triky areas, and a checkstyle fix. I should also mention that, with this approach, there's more overhead. This is true even when -x is not given - the behavior is compatible still though. calculation for snapshots are always performed. ContentSummary is bigger than before (since it holds more long fields) so we need to transfer a larger object. IMHO, since HDFS-4995 adds more control over the calculation, and we're not creating a lot of ContentSummary objects, the above 2 things should be fine.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 7m 21s trunk passed
          +1 compile 9m 3s trunk passed
          +1 checkstyle 1m 30s trunk passed
          +1 mvnsite 2m 49s trunk passed
          +1 mvneclipse 0m 41s trunk passed
          +1 findbugs 5m 28s trunk passed
          +1 javadoc 2m 18s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 2m 5s the patch passed
          +1 compile 7m 16s the patch passed
          +1 cc 7m 16s the patch passed
          +1 javac 7m 16s the patch passed
          +1 checkstyle 1m 37s root: The patch generated 0 new + 352 unchanged - 11 fixed = 352 total (was 363)
          +1 mvnsite 2m 29s the patch passed
          +1 mvneclipse 0m 44s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 5m 0s the patch passed
          +1 javadoc 2m 13s the patch passed
          +1 unit 8m 56s hadoop-common in the patch passed.
          +1 unit 1m 3s hadoop-hdfs-client in the patch passed.
          -1 unit 92m 23s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 34s The patch does not generate ASF License warnings.
          155m 24s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestLargeBlockReport
            hadoop.hdfs.TestReconstructStripedFile
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823173/HDFS-8986.04.patch
          JIRA Issue HDFS-8986
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml cc
          uname Linux 70bfeaecc96d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a428d4f
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16399/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16399/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16399/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 7m 21s trunk passed +1 compile 9m 3s trunk passed +1 checkstyle 1m 30s trunk passed +1 mvnsite 2m 49s trunk passed +1 mvneclipse 0m 41s trunk passed +1 findbugs 5m 28s trunk passed +1 javadoc 2m 18s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 5s the patch passed +1 compile 7m 16s the patch passed +1 cc 7m 16s the patch passed +1 javac 7m 16s the patch passed +1 checkstyle 1m 37s root: The patch generated 0 new + 352 unchanged - 11 fixed = 352 total (was 363) +1 mvnsite 2m 29s the patch passed +1 mvneclipse 0m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 5m 0s the patch passed +1 javadoc 2m 13s the patch passed +1 unit 8m 56s hadoop-common in the patch passed. +1 unit 1m 3s hadoop-hdfs-client in the patch passed. -1 unit 92m 23s hadoop-hdfs in the patch failed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 155m 24s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestLargeBlockReport   hadoop.hdfs.TestReconstructStripedFile Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823173/HDFS-8986.04.patch JIRA Issue HDFS-8986 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml cc uname Linux 70bfeaecc96d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a428d4f Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16399/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16399/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16399/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Hello Xiao Chen, thanks again for the new patch. It looks mostly good to me. One minor issue:

          The parameter of ContentSummary#Builder methods needs meaningful names. So instead of

          public Builder snapshotLength(long val) {
            this.snapshotLength = val;
          

          The parameter val should be named snapshotLength or simply length.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Hello Xiao Chen , thanks again for the new patch. It looks mostly good to me. One minor issue: The parameter of ContentSummary#Builder methods needs meaningful names. So instead of public Builder snapshotLength( long val) { this .snapshotLength = val; The parameter val should be named snapshotLength or simply length .
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Wei-Chiu, patch 5 attached to rename the params.

          Show
          xiaochen Xiao Chen added a comment - Thanks Wei-Chiu, patch 5 attached to rename the params.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Re-trigger the build.

          +1 pending Jenkins.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Re-trigger the build. +1 pending Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 1m 33s Maven dependency ordering for branch
          +1 mvninstall 6m 38s trunk passed
          +1 compile 6m 47s trunk passed
          +1 checkstyle 1m 29s trunk passed
          +1 mvnsite 2m 20s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 4m 27s trunk passed
          +1 javadoc 2m 0s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 54s the patch passed
          +1 compile 6m 42s the patch passed
          +1 cc 6m 42s the patch passed
          +1 javac 6m 42s the patch passed
          -0 checkstyle 1m 28s root: The patch generated 3 new + 352 unchanged - 11 fixed = 355 total (was 363)
          +1 mvnsite 2m 19s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 4m 49s the patch passed
          +1 javadoc 1m 59s the patch passed
          +1 unit 8m 9s hadoop-common in the patch passed.
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed.
          +1 unit 56m 35s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          113m 8s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824869/HDFS-8986.06.patch
          JIRA Issue HDFS-8986
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml cc
          uname Linux 8c9103869a99 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 115ecb5
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16499/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16499/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16499/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 1m 33s Maven dependency ordering for branch +1 mvninstall 6m 38s trunk passed +1 compile 6m 47s trunk passed +1 checkstyle 1m 29s trunk passed +1 mvnsite 2m 20s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 4m 27s trunk passed +1 javadoc 2m 0s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 54s the patch passed +1 compile 6m 42s the patch passed +1 cc 6m 42s the patch passed +1 javac 6m 42s the patch passed -0 checkstyle 1m 28s root: The patch generated 3 new + 352 unchanged - 11 fixed = 355 total (was 363) +1 mvnsite 2m 19s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 4m 49s the patch passed +1 javadoc 1m 59s the patch passed +1 unit 8m 9s hadoop-common in the patch passed. +1 unit 0m 54s hadoop-hdfs-client in the patch passed. +1 unit 56m 35s hadoop-hdfs in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 113m 8s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824869/HDFS-8986.06.patch JIRA Issue HDFS-8986 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml cc uname Linux 8c9103869a99 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 115ecb5 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16499/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16499/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16499/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 7 for pre-commit warnings.

          Show
          xiaochen Xiao Chen added a comment - Patch 7 for pre-commit warnings.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 8m 41s trunk passed
          +1 compile 8m 46s trunk passed
          +1 checkstyle 1m 34s trunk passed
          +1 mvnsite 2m 34s trunk passed
          +1 mvneclipse 0m 43s trunk passed
          +1 findbugs 4m 37s trunk passed
          +1 javadoc 2m 7s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 11s the patch passed
          +1 compile 6m 52s the patch passed
          +1 cc 6m 52s the patch passed
          +1 javac 6m 52s the patch passed
          -0 checkstyle 1m 28s root: The patch generated 4 new + 352 unchanged - 11 fixed = 356 total (was 363)
          +1 mvnsite 2m 19s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 4m 55s the patch passed
          +1 javadoc 2m 14s the patch passed
          +1 unit 8m 32s hadoop-common in the patch passed.
          +1 unit 1m 12s hadoop-hdfs-client in the patch passed.
          -1 unit 79m 42s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          141m 28s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestBootstrapStandby



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824903/HDFS-8986.07.patch
          JIRA Issue HDFS-8986
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml cc
          uname Linux 4d6251bf3a39 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dc7a1c5
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16500/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16500/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16500/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16500/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 8m 41s trunk passed +1 compile 8m 46s trunk passed +1 checkstyle 1m 34s trunk passed +1 mvnsite 2m 34s trunk passed +1 mvneclipse 0m 43s trunk passed +1 findbugs 4m 37s trunk passed +1 javadoc 2m 7s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 11s the patch passed +1 compile 6m 52s the patch passed +1 cc 6m 52s the patch passed +1 javac 6m 52s the patch passed -0 checkstyle 1m 28s root: The patch generated 4 new + 352 unchanged - 11 fixed = 356 total (was 363) +1 mvnsite 2m 19s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 55s the patch passed +1 javadoc 2m 14s the patch passed +1 unit 8m 32s hadoop-common in the patch passed. +1 unit 1m 12s hadoop-hdfs-client in the patch passed. -1 unit 79m 42s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 141m 28s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestBootstrapStandby Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824903/HDFS-8986.07.patch JIRA Issue HDFS-8986 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml cc uname Linux 4d6251bf3a39 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dc7a1c5 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16500/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16500/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16500/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16500/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Test failure looks unrelated and passed locally.
          Checkstyle warning I think can be ignored - it's more readable this way and hadoop builders are named this way in many fundamental places like acls, rpcs, etc.

          Show
          xiaochen Xiao Chen added a comment - Test failure looks unrelated and passed locally. Checkstyle warning I think can be ignored - it's more readable this way and hadoop builders are named this way in many fundamental places like acls, rpcs, etc.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          +1 committing this.

          Show
          jojochuang Wei-Chiu Chuang added a comment - +1 committing this.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10327 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10327/)
          HDFS-8986. Add option to -du to calculate directory space usage (weichiu: rev f0efea490e5aa9dd629d2199aae9c5b1290a17ee)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • (edit) hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCount.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Count.java
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ContentSummary.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10327 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10327/ ) HDFS-8986 . Add option to -du to calculate directory space usage (weichiu: rev f0efea490e5aa9dd629d2199aae9c5b1290a17ee) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java (edit) hadoop-common-project/hadoop-common/src/test/resources/testConf.xml (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCount.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Count.java (edit) hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ContentSummary.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Committed this to trunk. But the patch has an incompatible change in branch-2 due to HADOOP-11666. If I understand it correctly, the branch-2 patch should remove the output of spaceConsumed. But just to be sure, Xiao Chen could you rebase and make a branch-2 patch for a precommit check?

          Thanks.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Committed this to trunk. But the patch has an incompatible change in branch-2 due to HADOOP-11666 . If I understand it correctly, the branch-2 patch should remove the output of spaceConsumed. But just to be sure, Xiao Chen could you rebase and make a branch-2 patch for a precommit check? Thanks.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Wei-Chiu.

          HADOOP-11666 reverted an incompatible change, which caused backport conflicts on this patch. This patch itself is compatible IMO, attaching a patch For branch-2.

          Show
          xiaochen Xiao Chen added a comment - Thanks Wei-Chiu. HADOOP-11666 reverted an incompatible change, which caused backport conflicts on this patch. This patch itself is compatible IMO, attaching a patch For branch-2.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 7m 13s Docker failed to build yetus/hadoop:b59b8b7.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825068/HDFS-8986.branch-2.patch
          JIRA Issue HDFS-8986
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16513/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 7m 13s Docker failed to build yetus/hadoop:b59b8b7. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825068/HDFS-8986.branch-2.patch JIRA Issue HDFS-8986 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16513/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 1m 1s Maven dependency ordering for branch
          +1 mvninstall 6m 42s branch-2 passed
          +1 compile 5m 34s branch-2 passed with JDK v1.8.0_101
          +1 compile 6m 36s branch-2 passed with JDK v1.7.0_111
          +1 checkstyle 1m 29s branch-2 passed
          +1 mvnsite 2m 31s branch-2 passed
          +1 mvneclipse 0m 47s branch-2 passed
          +1 findbugs 5m 23s branch-2 passed
          +1 javadoc 2m 4s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 3m 0s branch-2 passed with JDK v1.7.0_111
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 2m 2s the patch passed
          +1 compile 6m 10s the patch passed with JDK v1.8.0_101
          +1 cc 6m 10s the patch passed
          +1 javac 6m 10s the patch passed
          +1 compile 6m 53s the patch passed with JDK v1.7.0_111
          +1 cc 6m 53s the patch passed
          +1 javac 6m 53s the patch passed
          -0 checkstyle 1m 29s root: The patch generated 4 new + 346 unchanged - 10 fixed = 350 total (was 356)
          +1 mvnsite 2m 29s the patch passed
          +1 mvneclipse 0m 45s the patch passed
          -1 whitespace 0m 0s The patch has 47 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 6m 10s the patch passed
          +1 javadoc 2m 5s the patch passed with JDK v1.8.0_101
          +1 javadoc 2m 58s the patch passed with JDK v1.7.0_111
          -1 unit 11m 0s hadoop-common in the patch failed with JDK v1.7.0_111.
          +1 unit 1m 15s hadoop-hdfs-client in the patch passed with JDK v1.7.0_111.
          -1 unit 78m 44s hadoop-hdfs in the patch failed with JDK v1.7.0_111.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          257m 52s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.server.namenode.TestNameNodeMXBean
            hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
            hadoop.hdfs.TestReplication
          JDK v1.8.0_101 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
            org.apache.hadoop.hdfs.TestLeaseRecovery2
          JDK v1.7.0_111 Failed junit tests hadoop.security.TestGroupsCaching
            hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.namenode.TestAuditLogger
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825068/HDFS-8986.branch-2.patch
          JIRA Issue HDFS-8986
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml cc
          uname Linux 05c27be85d90 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / e1d3b8e
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16516/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16516/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16516/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_111.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16516/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16516/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16516/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 1m 1s Maven dependency ordering for branch +1 mvninstall 6m 42s branch-2 passed +1 compile 5m 34s branch-2 passed with JDK v1.8.0_101 +1 compile 6m 36s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 1m 29s branch-2 passed +1 mvnsite 2m 31s branch-2 passed +1 mvneclipse 0m 47s branch-2 passed +1 findbugs 5m 23s branch-2 passed +1 javadoc 2m 4s branch-2 passed with JDK v1.8.0_101 +1 javadoc 3m 0s branch-2 passed with JDK v1.7.0_111 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 2s the patch passed +1 compile 6m 10s the patch passed with JDK v1.8.0_101 +1 cc 6m 10s the patch passed +1 javac 6m 10s the patch passed +1 compile 6m 53s the patch passed with JDK v1.7.0_111 +1 cc 6m 53s the patch passed +1 javac 6m 53s the patch passed -0 checkstyle 1m 29s root: The patch generated 4 new + 346 unchanged - 10 fixed = 350 total (was 356) +1 mvnsite 2m 29s the patch passed +1 mvneclipse 0m 45s the patch passed -1 whitespace 0m 0s The patch has 47 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 6m 10s the patch passed +1 javadoc 2m 5s the patch passed with JDK v1.8.0_101 +1 javadoc 2m 58s the patch passed with JDK v1.7.0_111 -1 unit 11m 0s hadoop-common in the patch failed with JDK v1.7.0_111. +1 unit 1m 15s hadoop-hdfs-client in the patch passed with JDK v1.7.0_111. -1 unit 78m 44s hadoop-hdfs in the patch failed with JDK v1.7.0_111. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 257m 52s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.namenode.TestNameNodeMXBean   hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages   hadoop.hdfs.TestReplication JDK v1.8.0_101 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle   org.apache.hadoop.hdfs.TestLeaseRecovery2 JDK v1.7.0_111 Failed junit tests hadoop.security.TestGroupsCaching   hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestAuditLogger   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825068/HDFS-8986.branch-2.patch JIRA Issue HDFS-8986 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml cc uname Linux 05c27be85d90 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / e1d3b8e Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16516/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16516/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16516/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_111.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16516/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16516/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16516/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Wei-Chiu Chuang for triggering a new run!

          • Checkstyle is same as trunk, more readable this way.
          • whitespace is false negative, not related.
          • All failed tests look unrelated, and passed locally (with jdk7).
          Show
          xiaochen Xiao Chen added a comment - Thanks Wei-Chiu Chuang for triggering a new run! Checkstyle is same as trunk, more readable this way. whitespace is false negative, not related. All failed tests look unrelated, and passed locally (with jdk7).
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          +1 on branch-2 patch. Test failures are not reproducible in my local tree (except for TestDataNodeHotSwapVolumes.testRemoveVolumeBeingWritten, which is a known flaky test)

          Committed to branch-2 and branch-2.8 Thanks Gautam Gopalakrishnan for reporting the bug, Jagadesh Kiran N, Harsh J, Chris Nauroth for commenting and Xiao Chen for the patch.

          Show
          jojochuang Wei-Chiu Chuang added a comment - +1 on branch-2 patch. Test failures are not reproducible in my local tree (except for TestDataNodeHotSwapVolumes.testRemoveVolumeBeingWritten, which is a known flaky test) Committed to branch-2 and branch-2.8 Thanks Gautam Gopalakrishnan for reporting the bug, Jagadesh Kiran N , Harsh J , Chris Nauroth for commenting and Xiao Chen for the patch.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Wei-Chiu Chuang for the reviews, and everyone for creating and thoughtful discussions on the issue.

          I set the fix versions according to Wei-Chiu's comment above.

          Show
          xiaochen Xiao Chen added a comment - Thanks Wei-Chiu Chuang for the reviews, and everyone for creating and thoughtful discussions on the issue. I set the fix versions according to Wei-Chiu's comment above.
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for the work, Xiao Chen. For the following code, doing the computation again may be too expensive. I think we only need to let computeContentSummary4Snapshot directly return summary.getCounts(), and we add it to summary.getCounts() and summary.getSnapshotCounts() separately.

          637	      // Also compute ContentSummary for snapshotCounts (So we can extract it
          638	      // later from the ContentSummary of all).
          639	      sf.computeContentSummary4Snapshot(summary.getBlockStoragePolicySuite(),
          640	          summary.getSnapshotCounts());
          
          Show
          jingzhao Jing Zhao added a comment - Thanks for the work, Xiao Chen . For the following code, doing the computation again may be too expensive. I think we only need to let computeContentSummary4Snapshot directly return summary.getCounts() , and we add it to summary.getCounts() and summary.getSnapshotCounts() separately. 637 // Also compute ContentSummary for snapshotCounts (So we can extract it 638 // later from the ContentSummary of all). 639 sf.computeContentSummary4Snapshot(summary.getBlockStoragePolicySuite(), 640 summary.getSnapshotCounts());
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Jing Zhao for the great comment! I created HDFS-10875 and attached a patch there.

          Show
          xiaochen Xiao Chen added a comment - Thanks Jing Zhao for the great comment! I created HDFS-10875 and attached a patch there.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Xiao Chen could you explain why The -x option is ignored if -u or -q option is given? Thx.

          Show
          xiaobingo Xiaobing Zhou added a comment - Xiao Chen could you explain why The -x option is ignored if -u or -q option is given? Thx.

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              ggop Gautam Gopalakrishnan
            • Votes:
              1 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development