Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-6857

FsShell should report raw disk usage including replication factor

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: fs
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      The output format of hadoop fs -du has been changed. It shows not only the file size but also the raw disk usage including the replication factor.

      Description

      Currently FsShell report HDFS usage with "hadoop fs -dus <path>" command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe "hadoop fs -dus -raw <path>"?). This will allow to assess resource usage more accurately. – Alex K

      1. HADOOP-6857.patch
        40 kB
        Byron Wong
      2. HADOOP-6857.patch
        11 kB
        Byron Wong
      3. HADOOP-6857.patch
        3 kB
        Byron Wong
      4. HADOOP-6857-revert.patch
        35 kB
        Byron Wong
      5. revert-HADOOP-6857-from-branch-2.patch
        42 kB
        Akira Ajisaka
      6. show-space-consumed.txt
        4 kB
        Aaron T. Myers

        Issue Links

          Activity

          Hide
          atm Aaron T. Myers added a comment -

          This patch adds a new column to the output of "hadoop fs -du" and "hadoop fs -dus" which shows the disk space consumed (file size * per-file replication factor) of the paths matched by these commands.

          Show
          atm Aaron T. Myers added a comment - This patch adds a new column to the output of "hadoop fs -du" and "hadoop fs -dus" which shows the disk space consumed (file size * per-file replication factor) of the paths matched by these commands.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12449749/show-space-consumed.txt
          against trunk revision 964993.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449749/show-space-consumed.txt against trunk revision 964993. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/console This message is automatically generated.
          Hide
          eli Eli Collins added a comment -

          Hey Aaron,

          Patch looks good. Mind creating o.a.h.fs.TestFsShell.java and adding a test that shows files with two different replication levels works? (might need to mock up the replication level). Also, please test that TestShell and TestHDFSCLI in HDFS still pass for sanity.

          Wrt to rationale I think this change is kosher since FileStatus#getReplication is not hdfs-specific.

          Marking the jira as an incompatible change since IIRC the FsShell is considered a public API.

          Thanks,
          Eli

          Show
          eli Eli Collins added a comment - Hey Aaron, Patch looks good. Mind creating o.a.h.fs.TestFsShell.java and adding a test that shows files with two different replication levels works? (might need to mock up the replication level). Also, please test that TestShell and TestHDFSCLI in HDFS still pass for sanity. Wrt to rationale I think this change is kosher since FileStatus#getReplication is not hdfs-specific. Marking the jira as an incompatible change since IIRC the FsShell is considered a public API. Thanks, Eli
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          We already have "fs -count <path>" which counts bytes including replications. Is it good enough?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - We already have "fs -count <path>" which counts bytes including replications. Is it good enough?
          Hide
          jghoman Jakob Homan added a comment -

          Canceling patch until Nicholas' comments are addressed.

          Show
          jghoman Jakob Homan added a comment - Canceling patch until Nicholas' comments are addressed.
          Hide
          atm Aaron T. Myers added a comment -

          We already have "fs -count <path>" which counts bytes including replications. Is it good enough?

          "fs -count <path>" is indeed sufficient. Feel free to close this ticket.

          Show
          atm Aaron T. Myers added a comment - We already have "fs -count <path>" which counts bytes including replications. Is it good enough? "fs -count <path>" is indeed sufficient. Feel free to close this ticket.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Closing this. Thanks.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Closing this. Thanks.
          Hide
          eli Eli Collins added a comment -

          Why do we have separate "dus" and "count" commands? They seem to duplicate each other. Since the CLI needs to be backwards compatible not suggesting we remove one, but perhaps we should update the hdfs_shell docs to make it clear that they display the same info if there are not significant differences.

          ~ $ hadoop fs -dus /user/eli
          hdfs://haus01.sf.cloudera.com:10020/user/eli	86183666860
          
          ~ $ hadoop fs -count /user/eli
                     7           51        86183666860 hdfs://haus01.sf.cloudera.com:10020/user/eli
          ~ $ 
          
          Show
          eli Eli Collins added a comment - Why do we have separate "dus" and "count" commands? They seem to duplicate each other. Since the CLI needs to be backwards compatible not suggesting we remove one, but perhaps we should update the hdfs_shell docs to make it clear that they display the same info if there are not significant differences. ~ $ hadoop fs -dus /user/eli hdfs: //haus01.sf.cloudera.com:10020/user/eli 86183666860 ~ $ hadoop fs -count /user/eli 7 51 86183666860 hdfs: //haus01.sf.cloudera.com:10020/user/eli ~ $
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          "fs -count" was introduced for counting name objects. The disk space column was added later on.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - "fs -count" was introduced for counting name objects. The disk space column was added later on.
          Hide
          knoguchi Koji Noguchi added a comment -

          A little confused. I thought "fs -count" shows hdfs usage same as "fs -du" at the thrid column.

          [knoguchi ~]$ hadoop dfs -dus /user/knoguchi
          hdfs://abc-nn1.com/user/knoguchi       2603203340273
          [knoguchi ~]$ hadoop dfs -count /user/knoguchi
                  1580        20624      2603203340273 hdfs://abc-nn1.com/user/knoguchi
          [knoguchi ~]$ 
          

          If quota is enabled on that dir and "-q" is passed, it would show the remaining raw space available.

          [knoguchi ~]$ hadoop dfs -count -q /user/knoguchi
                 50000           27796  13194139533312   5384528402193         1580        20624      2603203340273 hdfs://abc-nn1.com/user/knoguchi
          [knoguchi ~]$ 
          

          You can get the raw space usage then. (quota - raw_remaining).
          However this is only if you have quota enabled on that particular dir.

          Show
          knoguchi Koji Noguchi added a comment - A little confused. I thought "fs -count" shows hdfs usage same as "fs -du" at the thrid column. [knoguchi ~]$ hadoop dfs -dus /user/knoguchi hdfs://abc-nn1.com/user/knoguchi 2603203340273 [knoguchi ~]$ hadoop dfs -count /user/knoguchi 1580 20624 2603203340273 hdfs://abc-nn1.com/user/knoguchi [knoguchi ~]$ If quota is enabled on that dir and "-q" is passed, it would show the remaining raw space available. [knoguchi ~]$ hadoop dfs -count -q /user/knoguchi 50000 27796 13194139533312 5384528402193 1580 20624 2603203340273 hdfs://abc-nn1.com/user/knoguchi [knoguchi ~]$ You can get the raw space usage then. (quota - raw_remaining). However this is only if you have quota enabled on that particular dir .
          Hide
          knoguchi Koji Noguchi added a comment -

          I think this number(raw usage) would be helpful. Not sure whether this should be in -du or -count and by default or as an option.

          Show
          knoguchi Koji Noguchi added a comment - I think this number(raw usage) would be helpful. Not sure whether this should be in -du or -count and by default or as an option.
          Hide
          hq4ever Maxim Veksler added a comment -

          Is there an easy cli method of obtaining this information?

          If not then this can be a welcome feature in several use cases (Like file server based on HDFS)

          Show
          hq4ever Maxim Veksler added a comment - Is there an easy cli method of obtaining this information? If not then this can be a welcome feature in several use cases (Like file server based on HDFS)
          Hide
          eli Eli Collins added a comment -

          I agree with Koji, seeing raw disk space usage should be easy to get from the CLI, you shouldn't have to enable a quota on a directory to see it, eg see below that nothing indicates raw disk usage:

          ~ $ hadoop fs -mkdir dir
          ~ $ hadoop fs -put f3mb dir
          ~ $ hadoop fs -dus dir
          hdfs://haus01.sf.cloudera.com:10020/user/eli/dir	3145728
          ~ $ hadoop fs -count -q dir
                  none             inf            none             inf            1            1            3145728 hdfs://haus01.sf.cloudera.com:10020/user/eli/dir
          

          It also sounds like the original issue here was that disk space usage should not have been added to count if it was already available in dus. Even if we added raw disk usage to "count", it's wonky that a user can logical disk usage with "du" but has to switch over to a separate "count" command to get raw disk usage. Why not make all disk usage available from a single command?

          Sound reasonable Nicholas?

          Show
          eli Eli Collins added a comment - I agree with Koji, seeing raw disk space usage should be easy to get from the CLI, you shouldn't have to enable a quota on a directory to see it, eg see below that nothing indicates raw disk usage: ~ $ hadoop fs -mkdir dir ~ $ hadoop fs -put f3mb dir ~ $ hadoop fs -dus dir hdfs: //haus01.sf.cloudera.com:10020/user/eli/dir 3145728 ~ $ hadoop fs -count -q dir none inf none inf 1 1 3145728 hdfs: //haus01.sf.cloudera.com:10020/user/eli/dir It also sounds like the original issue here was that disk space usage should not have been added to count if it was already available in dus. Even if we added raw disk usage to "count", it's wonky that a user can logical disk usage with "du" but has to switch over to a separate "count" command to get raw disk usage. Why not make all disk usage available from a single command? Sound reasonable Nicholas?
          Hide
          Byron Wong Byron Wong added a comment -

          Updated Aaron T. Myers's patch to patch on top of trunk.
          I'd like to point out that creating a snapshot of a directory will increase the disk usage number reported by -du (-s). Is this what we want?

          Show
          Byron Wong Byron Wong added a comment - Updated Aaron T. Myers 's patch to patch on top of trunk. I'd like to point out that creating a snapshot of a directory will increase the disk usage number reported by -du (-s). Is this what we want?
          Hide
          shv Konstantin Shvachko added a comment -

          Byron, could you please clarify how -du reporting is different in the presence of snapshots?

          Show
          shv Konstantin Shvachko added a comment - Byron, could you please clarify how -du reporting is different in the presence of snapshots?
          Hide
          Byron Wong Byron Wong added a comment -

          In the case when a directory /D and snapshot S are in the exact same state (e.g. a fresh snapshot has been made), everything works fine, meaning the sum of the disk consumed numbers reported by -du /D equals the disk consumed number reported by -du -s /D.
          When /D and S start deviating (files getting renamed, deleted, etc.), the disk consumed calculation will take the lastFileSize within the snapshots, find the maximum replication factor for that file within the snapshots, multiply the 2 together, and increment disk consumed by that number, which inflates the total disk consumed calculation, so -du -s /D > the sum of numbers in -du /D.

          I'd also like to point out that this implementation only takes replication factor of a file into account, even if that replication factor is greater than number of data nodes, which further inflates the -du calculation. For example, if we setrep 10 a file when we only have 3 datanodes, -du will still multiply fileLength * 10, and report that number.

          Show
          Byron Wong Byron Wong added a comment - In the case when a directory /D and snapshot S are in the exact same state (e.g. a fresh snapshot has been made), everything works fine, meaning the sum of the disk consumed numbers reported by -du /D equals the disk consumed number reported by -du -s /D. When /D and S start deviating (files getting renamed, deleted, etc.), the disk consumed calculation will take the lastFileSize within the snapshots, find the maximum replication factor for that file within the snapshots, multiply the 2 together, and increment disk consumed by that number, which inflates the total disk consumed calculation, so -du -s /D > the sum of numbers in -du /D. I'd also like to point out that this implementation only takes replication factor of a file into account, even if that replication factor is greater than number of data nodes, which further inflates the -du calculation. For example, if we setrep 10 a file when we only have 3 datanodes, -du will still multiply fileLength * 10, and report that number.
          Hide
          shv Konstantin Shvachko added a comment -

          Byron, Could you post some examples (printouts) in support of your use case.

          if we setrep 10 a file when we only have 3 datanodes, -du will still multiply fileLength * 10

          This seems to be consistent with how quotas are calculated. DU should report numbers consistent with the quotas. We can change diskSpaceConsumed() if people want more precise usage info both for du and quotas, but at this point it will be a change in semantics. Let's keep it the current way.

          Show
          shv Konstantin Shvachko added a comment - Byron, Could you post some examples (printouts) in support of your use case. if we setrep 10 a file when we only have 3 datanodes, -du will still multiply fileLength * 10 This seems to be consistent with how quotas are calculated. DU should report numbers consistent with the quotas. We can change diskSpaceConsumed() if people want more precise usage info both for du and quotas, but at this point it will be a change in semantics. Let's keep it the current way.
          Hide
          Byron Wong Byron Wong added a comment -

          Scenario 1: "/test" is a snapshottable directory with a file "a" that has 41 bytes, replication factor 3.
          We run hadoop fs -du /test:

          41  123  /test/a
          

          which is consistent with what we get when we run hadoop fs -du -s /test:

          41  123  /test
          

          When we create a snapshot "ss1" and rerun the -du commands again, we still get the same results as seen above.

          Let's say we now run hadoop fs -mv /test/a /test/b.
          Now, when we run hadoop fs -du /test, we get:

          41  123  /test/b
          

          which is inconsistent with what we see when we run hadoop fs -du -s /test:

          41  246  /test
          

          If we report this process again (i.e. create snapshot, rename /test/b to /test/a), we get more and more deviations between the 2 commands.

          Show
          Byron Wong Byron Wong added a comment - Scenario 1 : "/test" is a snapshottable directory with a file "a" that has 41 bytes, replication factor 3. We run hadoop fs -du /test : 41 123 /test/a which is consistent with what we get when we run hadoop fs -du -s /test : 41 123 /test When we create a snapshot "ss1" and rerun the -du commands again, we still get the same results as seen above. Let's say we now run hadoop fs -mv /test/a /test/b . Now, when we run hadoop fs -du /test , we get: 41 123 /test/b which is inconsistent with what we see when we run hadoop fs -du -s /test : 41 246 /test If we report this process again (i.e. create snapshot, rename /test/b to /test/a), we get more and more deviations between the 2 commands.
          Hide
          Byron Wong Byron Wong added a comment -

          Scenario 2: we still have "snapshottable directory "/test" with same file "a". We then create a fresh snapshot "ss1". We then run hadoop fs -rm -skipTrash /test/a.
          hadoop fs -du /test gives an empty output, as expected.
          hadoop fs -du -s /test outputs:

          41  123  /test
          

          which makes sense, given that we know about the existence of the snapshot.
          However, when we run hadoop fs -du -s /test/.snapshot/ss1, we get:

          0  0  /test/.snapshot/ss1
          

          This is inconsistent with the numbers we get when we run hadoop fs -du /test/.snapshot/ss1:

          41  123  /test/.snapshot/ss1/a
          

          Upon further investigation, we see that running hadoop fs -du -s /test/.snapshot/anySnapshot gives us the information about the current state of the real directory. This means that hadoop fs -du -s /test/.snapshot/anySnapshot is equivalent to running hadoop fs -du /test/ and adding the numbers up, which is non-intuitive.
          For example, let's add a 2 byte, 3 replication file "/test/1" (/test/a is still deleted). Now hadoop fs -du -s /test/.snapshot/ss1 gives us:

          2  6  /test/.snapshot/ss1
          

          whereas the results of hadoop fs -du /test/.snapshot/ss1 remains the same:

          41  123  /test/.snapshot/ss1/a
          
          Show
          Byron Wong Byron Wong added a comment - Scenario 2 : we still have "snapshottable directory "/test" with same file "a". We then create a fresh snapshot "ss1". We then run hadoop fs -rm -skipTrash /test/a . hadoop fs -du /test gives an empty output, as expected. hadoop fs -du -s /test outputs: 41 123 /test which makes sense, given that we know about the existence of the snapshot. However, when we run hadoop fs -du -s /test/.snapshot/ss1 , we get: 0 0 /test/.snapshot/ss1 This is inconsistent with the numbers we get when we run hadoop fs -du /test/.snapshot/ss1 : 41 123 /test/.snapshot/ss1/a Upon further investigation, we see that running hadoop fs -du -s /test/.snapshot/anySnapshot gives us the information about the current state of the real directory. This means that hadoop fs -du -s /test/.snapshot/anySnapshot is equivalent to running hadoop fs -du /test/ and adding the numbers up, which is non-intuitive. For example, let's add a 2 byte, 3 replication file "/test/1" (/test/a is still deleted). Now hadoop fs -du -s /test/.snapshot/ss1 gives us: 2 6 /test/.snapshot/ss1 whereas the results of hadoop fs -du /test/.snapshot/ss1 remains the same: 41 123 /test/.snapshot/ss1/a
          Hide
          shv Konstantin Shvachko added a comment -

          Scenario 1 looks rather unusual, but again this is how quotas are counted: When you modify a file in a snapshot, then it is counted twice for quotas. Even if it doesn't actually store axtra blocks. If people want to change this, it should be a different jira related to qoutas. I don't think anything needs to be fixed in this regard here.
          Scenario 2 indeed gives inconsistent results for du and dus, because du resolves path to point to the current tree rather than to the snapshot contents. This doesn't look right to me.

          Show
          shv Konstantin Shvachko added a comment - Scenario 1 looks rather unusual, but again this is how quotas are counted: When you modify a file in a snapshot, then it is counted twice for quotas. Even if it doesn't actually store axtra blocks. If people want to change this, it should be a different jira related to qoutas. I don't think anything needs to be fixed in this regard here. Scenario 2 indeed gives inconsistent results for du and dus , because du resolves path to point to the current tree rather than to the snapshot contents. This doesn't look right to me.
          Hide
          Byron Wong Byron Wong added a comment -

          Attached new patch.
          This patch addresses Scenario 2. Snapshot$Root should computeContentSummary based on its snapshotId rather than the current state. Added unit test to verify this case.

          Show
          Byron Wong Byron Wong added a comment - Attached new patch. This patch addresses Scenario 2. Snapshot$Root should computeContentSummary based on its snapshotId rather than the current state. Added unit test to verify this case.
          Hide
          shv Konstantin Shvachko added a comment -

          This looks good. +1 from me.
          The test indeed failes on trunk and succeeds with the patch.
          Let's check with Jenkins.

          Show
          shv Konstantin Shvachko added a comment - This looks good. +1 from me. The test indeed failes on trunk and succeeds with the patch. Let's check with Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12676380/HADOOP-6857.patch
          against trunk revision db45f04.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.web.TestWebHDFSForHA
          org.apache.hadoop.cli.TestHDFSCLI

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4942//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4942//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676380/HADOOP-6857.patch against trunk revision db45f04. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.web.TestWebHDFSForHA org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4942//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4942//console This message is automatically generated.
          Hide
          shv Konstantin Shvachko added a comment -

          Looks like format change of du reporting broke TestHDFSCLI.
          Not sure about TestWebHDFSForHA.

          Show
          shv Konstantin Shvachko added a comment - Looks like format change of du reporting broke TestHDFSCLI. Not sure about TestWebHDFSForHA.
          Hide
          Byron Wong Byron Wong added a comment -

          Added new patch to fix TestHDFSCLI.
          I don't think the failure in TestWebHDFSFOrHA has to do with my changes. It passed locally for me.

          Show
          Byron Wong Byron Wong added a comment - Added new patch to fix TestHDFSCLI. I don't think the failure in TestWebHDFSFOrHA has to do with my changes. It passed locally for me.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12676965/HADOOP-6857.patch
          against trunk revision e2be333.

          -1 patch. Trunk compilation may be broken.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4948//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676965/HADOOP-6857.patch against trunk revision e2be333. -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4948//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12676965/HADOOP-6857.patch
          against trunk revision a52eb4b.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4950//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4950//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676965/HADOOP-6857.patch against trunk revision a52eb4b. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4950//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4950//console This message is automatically generated.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6345 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6345/)
          HADOOP-6857. FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          • hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6345 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6345/ ) HADOOP-6857 . FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml hadoop-common-project/hadoop-common/src/test/resources/testConf.xml hadoop-common-project/hadoop-common/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          Hide
          shv Konstantin Shvachko added a comment -

          I just committed this. Congratulations Byron.

          Show
          shv Konstantin Shvachko added a comment - I just committed this. Congratulations Byron.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #724 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/724/)
          HADOOP-6857. FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #724 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/724/ ) HADOOP-6857 . FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/resources/testConf.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1913 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1913/)
          HADOOP-6857. FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1913 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1913/ ) HADOOP-6857 . FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/resources/testConf.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1938 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1938/)
          HADOOP-6857. FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1938 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1938/ ) HADOOP-6857 . FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-common-project/hadoop-common/src/test/resources/testConf.xml hadoop-common-project/hadoop-common/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          Hide
          ajisakaa Akira Ajisaka added a comment -

          This change is incompatible since it changes the output of the CLI command. (http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/Compatibility.html#Command_Line_Interface_CLI)
          Hi Konstantin Shvachko, should we revert this change from branch-2?

          Show
          ajisakaa Akira Ajisaka added a comment - This change is incompatible since it changes the output of the CLI command. ( http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/Compatibility.html#Command_Line_Interface_CLI ) Hi Konstantin Shvachko , should we revert this change from branch-2?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          We still need to do either of the below things

          • Move the section in CHANGES.txt to incompatible change and keep the fix in branch-2.
          • Revert the commit from branch-2.

          I like the idea of reverting the change.

          Show
          ajisakaa Akira Ajisaka added a comment - We still need to do either of the below things Move the section in CHANGES.txt to incompatible change and keep the fix in branch-2. Revert the commit from branch-2. I like the idea of reverting the change.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Prepared a patch to revert the commit. I'd like to revert it in this weekend if there is at least one +1 or no comment. Any comments are welcome.

          Show
          ajisakaa Akira Ajisaka added a comment - Prepared a patch to revert the commit. I'd like to revert it in this weekend if there is at least one +1 or no comment. Any comments are welcome.
          Hide
          shv Konstantin Shvachko added a comment -

          Sorry missed your comment 2 days ago. Let me look at it.

          Show
          shv Konstantin Shvachko added a comment - Sorry missed your comment 2 days ago. Let me look at it.
          Hide
          shv Konstantin Shvachko added a comment -

          So there is a bug fix here and the output format change. The latter is incompatible for branch 2. I think we should revert the format change, but leave the bug fix. Byron, could you please confirm. Akira, does it make sense to you?

          Show
          shv Konstantin Shvachko added a comment - So there is a bug fix here and the output format change. The latter is incompatible for branch 2. I think we should revert the format change, but leave the bug fix. Byron, could you please confirm. Akira, does it make sense to you?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Make sense. We should leave the bug fix.

          Show
          ajisakaa Akira Ajisaka added a comment - Make sense. We should leave the bug fix.
          Hide
          Byron Wong Byron Wong added a comment -

          Yea, I agree. The bug fix should remain, but the output changes can be reverted.

          Show
          Byron Wong Byron Wong added a comment - Yea, I agree. The bug fix should remain, but the output changes can be reverted.
          Hide
          Byron Wong Byron Wong added a comment -

          Attached HADOOP-6857-revert.patch.
          Reverted only the changes that involved command line output.

          Show
          Byron Wong Byron Wong added a comment - Attached HADOOP-6857 -revert.patch. Reverted only the changes that involved command line output.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Byron Wong for the patch. By the way, can we revert the incompatible change in a separate jira?
          Release note automatically collects information from JIRA (http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/releasenotes.html), so I'd like to make them as follows:

          • HADOOP-6857: Fix version -> 3.0.0 and mark as incompatible change.
          • A separate jira that reverts incompatible change: Fix version -> 2.7.0 (commit branch-2 only)
          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Byron Wong for the patch. By the way, can we revert the incompatible change in a separate jira? Release note automatically collects information from JIRA ( http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/releasenotes.html ), so I'd like to make them as follows: HADOOP-6857 : Fix version -> 3.0.0 and mark as incompatible change. A separate jira that reverts incompatible change: Fix version -> 2.7.0 (commit branch-2 only)
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Filed HADOOP-11666.

          Show
          ajisakaa Akira Ajisaka added a comment - Filed HADOOP-11666 .
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Moved the section to 3.0.0 in CHANGES.txt. Thanks Byron Wong and Konstantin Shvachko!

          Show
          ajisakaa Akira Ajisaka added a comment - Moved the section to 3.0.0 in CHANGES.txt. Thanks Byron Wong and Konstantin Shvachko !
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7251 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7251/)
          Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7251 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7251/ ) Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #122 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/122/)
          Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #122 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/122/ ) Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #856 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/856/)
          Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #856 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/856/ ) Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2054 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2054/)
          Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2054 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2054/ ) Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #113 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/113/)
          Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #113 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/113/ ) Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #122 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/122/)
          Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #122 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/122/ ) Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2072 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2072/)
          Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2072 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2072/ ) Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          aw Allen Wittenauer added a comment -

          This is still sitting in the changes.txt file for branch-2 and probably 2.7....

          Show
          aw Allen Wittenauer added a comment - This is still sitting in the changes.txt file for branch-2 and probably 2.7....
          Hide
          shv Konstantin Shvachko added a comment -

          Because it was committed to branch-2 and 2.7. As discussed earlier there is bug fix, which went everywhere, and the print format change, which was reverted from branch 2 by HADOOP-11666.

          Show
          shv Konstantin Shvachko added a comment - Because it was committed to branch-2 and 2.7. As discussed earlier there is bug fix, which went everywhere, and the print format change, which was reverted from branch 2 by HADOOP-11666 .

            People

            • Assignee:
              Byron Wong Byron Wong
              Reporter:
              alexvk Alex Kozlov
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development