Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9038

DFS reserved space is erroneously counted towards non-DFS used.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha2
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-5215 changed the DataNode volume available space calculation to consider the reserved space held by the dfs.datanode.du.reserved configuration property. As a side effect, reserved space is now counted towards non-DFS used. I don't believe it was intentional to change the definition of non-DFS used. This issue proposes restoring the prior behavior: do not count reserved space towards non-DFS used.

      1. GetFree.java
        0.4 kB
        Arpit Agarwal
      2. HDFS-9038.patch
        25 kB
        Brahma Reddy Battula
      3. HDFS-9038-002.patch
        25 kB
        Brahma Reddy Battula
      4. HDFS-9038-003.patch
        25 kB
        Brahma Reddy Battula
      5. HDFS-9038-004.patch
        26 kB
        Brahma Reddy Battula
      6. HDFS-9038-005.patch
        29 kB
        Brahma Reddy Battula
      7. HDFS-9038-006.patch
        29 kB
        Brahma Reddy Battula
      8. HDFS-9038-007.patch
        30 kB
        Brahma Reddy Battula
      9. HDFS-9038-008.patch
        30 kB
        Brahma Reddy Battula
      10. HDFS-9038-009.patch
        31 kB
        Brahma Reddy Battula
      11. HDFS-9038-010.patch
        31 kB
        Brahma Reddy Battula
      12. HDFS-9038-011.patch
        32 kB
        Brahma Reddy Battula
      13. HDFS-9038-branch-2.7.patch
        32 kB
        Brahma Reddy Battula

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          I'd appreciate if some of the participants on HDFS-5215 could chime in with their opinions. cc Brahma Reddy Battula, Uma Maheswara Rao G, Yongjun Zhang and Kihwal Lee. Thank you.

          Show
          cnauroth Chris Nauroth added a comment - I'd appreciate if some of the participants on HDFS-5215 could chime in with their opinions. cc Brahma Reddy Battula , Uma Maheswara Rao G , Yongjun Zhang and Kihwal Lee . Thank you.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          2 ways we can solve this,
          1) passing non-dfsused as a metric directly from DN side itself, similar to other stats, i.e. Moving the calculation to DN side
          2) passing 'reserved' values and using that to calculate non-dfs usage at NN side.

          Show
          brahmareddy Brahma Reddy Battula added a comment - 2 ways we can solve this, 1) passing non-dfsused as a metric directly from DN side itself, similar to other stats, i.e. Moving the calculation to DN side 2) passing 'reserved' values and using that to calculate non-dfs usage at NN side.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Reserved is not updated in the DatanodeDescriptor so need to propogate back to NN side via heartbeat if we want to update that..So We can solve it by anyone of above apporach..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Reserved is not updated in the DatanodeDescriptor so need to propogate back to NN side via heartbeat if we want to update that..So We can solve it by anyone of above apporach..
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks brahma for the options.
          #1 would be better IMO.

          Show
          vinayrpet Vinayakumar B added a comment - Thanks brahma for the options. #1 would be better IMO.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Moving out all non-critical / non-blocker issues that didn't make it out of 2.7.2 into 2.7.3.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Moving out all non-critical / non-blocker issues that didn't make it out of 2.7.2 into 2.7.3.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Uploading the patch to makes nonDFSUsed accurate by considering reservedForReplicas also.Actually bug is not there as mentioned in description,Since reserved was already subtracted from total capacity, nonDfsUsed was not including the reserved.

          But getAvailable() value was calculated directly from DF, but there reserved was not considered, which was fixed.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Uploading the patch to makes nonDFSUsed accurate by considering reservedForReplicas also.Actually bug is not there as mentioned in description,Since reserved was already subtracted from total capacity, nonDfsUsed was not including the reserved. But getAvailable() value was calculated directly from DF, but there reserved was not considered, which was fixed.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 4m 12s trunk passed
          +1 compile 1m 20s trunk passed with JDK v1.8.0_66
          +1 compile 1m 19s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 24s trunk passed
          +1 mvneclipse 0m 35s trunk passed
          -1 findbugs 2m 14s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 1m 52s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 35s trunk passed with JDK v1.7.0_79
          -1 mvninstall 0m 29s hadoop-hdfs in the patch failed.
          +1 compile 1m 14s the patch passed with JDK v1.8.0_66
          +1 cc 1m 14s the patch passed
          +1 javac 1m 14s the patch passed
          +1 compile 1m 9s the patch passed with JDK v1.7.0_79
          +1 cc 1m 9s the patch passed
          +1 javac 1m 9s the patch passed
          -1 checkstyle 0m 23s Patch generated 17 new checkstyle issues in hadoop-hdfs-project (total was 293, now 308).
          +1 mvneclipse 0m 31s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 40s the patch passed
          +1 javadoc 1m 45s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 38s the patch passed with JDK v1.7.0_79
          -1 unit 71m 52s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 67m 5s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79.
          -1 asflicense 0m 19s Patch generated 56 ASF License warnings.
          174m 47s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.TestAclsEndToEnd
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestHDFSFileSystemContract



          Subsystem Report/Notes
          Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-04
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770574/HDFS-9038.patch
          JIRA Issue HDFS-9038
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile cc
          uname Linux b8b2da95b5cb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build@2/patchprocess/apache-yetus-d0f6847/precommit/personality/hadoop.sh
          git revision trunk / 73b9c7b
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13381/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
          Max memory used 225MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13381/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 4m 12s trunk passed +1 compile 1m 20s trunk passed with JDK v1.8.0_66 +1 compile 1m 19s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 24s trunk passed +1 mvneclipse 0m 35s trunk passed -1 findbugs 2m 14s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 52s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 35s trunk passed with JDK v1.7.0_79 -1 mvninstall 0m 29s hadoop-hdfs in the patch failed. +1 compile 1m 14s the patch passed with JDK v1.8.0_66 +1 cc 1m 14s the patch passed +1 javac 1m 14s the patch passed +1 compile 1m 9s the patch passed with JDK v1.7.0_79 +1 cc 1m 9s the patch passed +1 javac 1m 9s the patch passed -1 checkstyle 0m 23s Patch generated 17 new checkstyle issues in hadoop-hdfs-project (total was 293, now 308). +1 mvneclipse 0m 31s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 40s the patch passed +1 javadoc 1m 45s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 38s the patch passed with JDK v1.7.0_79 -1 unit 71m 52s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 67m 5s hadoop-hdfs in the patch failed with JDK v1.7.0_79. +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79. -1 asflicense 0m 19s Patch generated 56 ASF License warnings. 174m 47s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.TestAclsEndToEnd JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestHDFSFileSystemContract Subsystem Report/Notes Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-04 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770574/HDFS-9038.patch JIRA Issue HDFS-9038 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile cc uname Linux b8b2da95b5cb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build@2/patchprocess/apache-yetus-d0f6847/precommit/personality/hadoop.sh git revision trunk / 73b9c7b Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13381/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13381/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 225MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13381/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Uploaded the patch fix the checkstyle issues.. Testcase failures are unrelated..Compilation errors are strange..? Chris Nauroth/Vinayakumar B kindly review..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Uploaded the patch fix the checkstyle issues.. Testcase failures are unrelated..Compilation errors are strange..? Chris Nauroth / Vinayakumar B kindly review..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 4m 21s trunk passed
          +1 compile 1m 20s trunk passed with JDK v1.8.0_66
          +1 compile 1m 14s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 27s trunk passed
          +1 mvneclipse 0m 35s trunk passed
          -1 findbugs 2m 24s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 1m 58s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 56s trunk passed with JDK v1.7.0_79
          -1 mvninstall 0m 33s hadoop-hdfs in the patch failed.
          +1 compile 1m 37s the patch passed with JDK v1.8.0_66
          +1 cc 1m 37s the patch passed
          +1 javac 1m 37s the patch passed
          +1 compile 1m 20s the patch passed with JDK v1.7.0_79
          +1 cc 1m 20s the patch passed
          +1 javac 1m 20s the patch passed
          -1 checkstyle 0m 25s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 293, now 295).
          +1 mvneclipse 0m 31s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 5m 4s the patch passed
          +1 javadoc 2m 2s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 58s the patch passed with JDK v1.7.0_79
          -1 unit 75m 10s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 69m 30s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79.
          -1 asflicense 0m 21s Patch generated 56 ASF License warnings.
          183m 44s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestBlockStoragePolicy
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.namenode.TestDecommissioningStatus



          Subsystem Report/Notes
          Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-05
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770754/HDFS-9038-002.patch
          JIRA Issue HDFS-9038
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile cc
          uname Linux 2224019fc6e8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-e8bd3ad/precommit/personality/hadoop.sh
          git revision trunk / 5667129
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13397/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13397/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 4m 21s trunk passed +1 compile 1m 20s trunk passed with JDK v1.8.0_66 +1 compile 1m 14s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 27s trunk passed +1 mvneclipse 0m 35s trunk passed -1 findbugs 2m 24s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 58s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 56s trunk passed with JDK v1.7.0_79 -1 mvninstall 0m 33s hadoop-hdfs in the patch failed. +1 compile 1m 37s the patch passed with JDK v1.8.0_66 +1 cc 1m 37s the patch passed +1 javac 1m 37s the patch passed +1 compile 1m 20s the patch passed with JDK v1.7.0_79 +1 cc 1m 20s the patch passed +1 javac 1m 20s the patch passed -1 checkstyle 0m 25s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 293, now 295). +1 mvneclipse 0m 31s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 4s the patch passed +1 javadoc 2m 2s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 58s the patch passed with JDK v1.7.0_79 -1 unit 75m 10s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 69m 30s hadoop-hdfs in the patch failed with JDK v1.7.0_79. +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79. -1 asflicense 0m 21s Patch generated 56 ASF License warnings. 183m 44s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.namenode.TestDecommissioningStatus Subsystem Report/Notes Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-05 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770754/HDFS-9038-002.patch JIRA Issue HDFS-9038 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile cc uname Linux 2224019fc6e8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-e8bd3ad/precommit/personality/hadoop.sh git revision trunk / 5667129 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13397/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13397/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13397/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          -1 mvninstall 0m 33s hadoop-hdfs in the patch failed.

          .

          Compile done on parent but mvn install done only on hadoop-hdfs.Allen Wittenauer can you please take look on this..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - -1 mvninstall 0m 33s hadoop-hdfs in the patch failed. . Compile done on parent but mvn install done only on hadoop-hdfs. Allen Wittenauer can you please take look on this..?
          Hide
          aw Allen Wittenauer added a comment -

          Yup. Working as intended.

          Show
          aw Allen Wittenauer added a comment - Yup. Working as intended.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Yup. Working as intended.

          IMO, since the patch contains multi-module changes (hadoop-hdfs-client and hadoop-hdfs),
          either "mvn install" should be done on parent directory, same as "mvn test-compile",
          or, should be executed in dependency order (hadoop-hdfs-client and hadoop-hdfs).

          Right?

          Show
          vinayrpet Vinayakumar B added a comment - Yup. Working as intended. IMO, since the patch contains multi-module changes (hadoop-hdfs-client and hadoop-hdfs), either "mvn install" should be done on parent directory, same as "mvn test-compile", or, should be executed in dependency order (hadoop-hdfs-client and hadoop-hdfs). Right?
          Hide
          vinayrpet Vinayakumar B added a comment -

          same case happened with HDFS-9057

          Show
          vinayrpet Vinayakumar B added a comment - same case happened with HDFS-9057
          Hide
          aw Allen Wittenauer added a comment -

          should be executed in dependency order (hadoop-hdfs-client and hadoop-hdfs).

          It's being done in the dependency order as dictated by the Yetus hadoop personality. If you feel the ordering is wrong, patches accepted.

          Show
          aw Allen Wittenauer added a comment - should be executed in dependency order (hadoop-hdfs-client and hadoop-hdfs). It's being done in the dependency order as dictated by the Yetus hadoop personality. If you feel the ordering is wrong, patches accepted.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          YETUS-170 raised to handle mvn install issue..can somebody review patch now..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - YETUS-170 raised to handle mvn install issue..can somebody review patch now..?
          Hide
          cnauroth Chris Nauroth added a comment -

          Brahma Reddy Battula, thank you for working on this. My earliest opportunity to code review this will be next Tuesday, 11/17. Of course, if someone else in the community would like to take up the review, that's great too.

          Show
          cnauroth Chris Nauroth added a comment - Brahma Reddy Battula , thank you for working on this. My earliest opportunity to code review this will be next Tuesday, 11/17. Of course, if someone else in the community would like to take up the review, that's great too.
          Hide
          cnauroth Chris Nauroth added a comment -

          Brahma Reddy Battula, thank you for the patch. This looks pretty good to me, though I'd also appreciate a second code review, maybe from Vinayakumar B or Arpit Agarwal.

          I just have one point of feedback right now. In DatanodeStorageInfo#toStorageReport, should the call to the StorageReport constructor pass along nonDfsUsed instead of hard-coding 0L?

          After that's addressed, I'll put this through some manual testing too.

          Show
          cnauroth Chris Nauroth added a comment - Brahma Reddy Battula , thank you for the patch. This looks pretty good to me, though I'd also appreciate a second code review, maybe from Vinayakumar B or Arpit Agarwal . I just have one point of feedback right now. In DatanodeStorageInfo#toStorageReport , should the call to the StorageReport constructor pass along nonDfsUsed instead of hard-coding 0L ? After that's addressed, I'll put this through some manual testing too.
          Hide
          vinayrpet Vinayakumar B added a comment -

          In DatanodeStorageInfo#toStorageReport, should the call to the StorageReport constructor pass along nonDfsUsed instead of hard-coding 0L?

          That was a good catch. Chris Nauroth.
          Apart from that everything else looks good to me as well. +1 once addressed.

          Show
          vinayrpet Vinayakumar B added a comment - In DatanodeStorageInfo#toStorageReport, should the call to the StorageReport constructor pass along nonDfsUsed instead of hard-coding 0L? That was a good catch. Chris Nauroth . Apart from that everything else looks good to me as well. +1 once addressed.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Chris Nauroth and Vinayakumar B thanks a for your reviews..uploaded the patch to address above comment.Kindly review..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Chris Nauroth and Vinayakumar B thanks a for your reviews..uploaded the patch to address above comment.Kindly review..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 8m 45s trunk passed
          +1 compile 2m 12s trunk passed with JDK v1.8.0_66
          +1 compile 1m 49s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 1m 43s trunk passed
          +1 mvneclipse 0m 31s trunk passed
          +1 findbugs 4m 27s trunk passed
          +1 javadoc 1m 48s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 37s trunk passed with JDK v1.7.0_79
          -1 mvninstall 0m 32s hadoop-hdfs in the patch failed.
          +1 compile 2m 8s the patch passed with JDK v1.8.0_66
          +1 cc 2m 8s the patch passed
          -1 javac 9m 52s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 49).
          +1 javac 2m 8s the patch passed
          +1 compile 1m 49s the patch passed with JDK v1.7.0_79
          +1 cc 1m 49s the patch passed
          -1 javac 11m 41s hadoop-hdfs-project-jdk1.7.0_79 with JDK v1.7.0_79 generated 3 new issues (was 51, now 51).
          +1 javac 1m 49s the patch passed
          -1 checkstyle 0m 25s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 293, now 295).
          +1 mvnsite 1m 42s the patch passed
          +1 mvneclipse 0m 30s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 4m 53s the patch passed
          +1 javadoc 1m 50s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 44s the patch passed with JDK v1.7.0_79
          -1 unit 76m 2s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 1m 10s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 78m 14s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          +1 unit 1m 8s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79.
          -1 asflicense 0m 24s Patch generated 56 ASF License warnings.
          201m 59s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSPermission
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
            hadoop.hdfs.qjournal.client.TestQuorumJournalManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:date2015-11-20
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773455/HDFS-9038-003.patch
          JIRA Issue HDFS-9038
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 2ccc37d6f170 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh
          git revision trunk / 4539131
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
          javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt
          javac hadoop-hdfs-project-jdk1.7.0_79: https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_79.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13575/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
          Max memory used 78MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13575/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 8m 45s trunk passed +1 compile 2m 12s trunk passed with JDK v1.8.0_66 +1 compile 1m 49s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 24s trunk passed +1 mvnsite 1m 43s trunk passed +1 mvneclipse 0m 31s trunk passed +1 findbugs 4m 27s trunk passed +1 javadoc 1m 48s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 37s trunk passed with JDK v1.7.0_79 -1 mvninstall 0m 32s hadoop-hdfs in the patch failed. +1 compile 2m 8s the patch passed with JDK v1.8.0_66 +1 cc 2m 8s the patch passed -1 javac 9m 52s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 49). +1 javac 2m 8s the patch passed +1 compile 1m 49s the patch passed with JDK v1.7.0_79 +1 cc 1m 49s the patch passed -1 javac 11m 41s hadoop-hdfs-project-jdk1.7.0_79 with JDK v1.7.0_79 generated 3 new issues (was 51, now 51). +1 javac 1m 49s the patch passed -1 checkstyle 0m 25s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 293, now 295). +1 mvnsite 1m 42s the patch passed +1 mvneclipse 0m 30s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 4m 53s the patch passed +1 javadoc 1m 50s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 44s the patch passed with JDK v1.7.0_79 -1 unit 76m 2s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 1m 10s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 78m 14s hadoop-hdfs in the patch failed with JDK v1.7.0_79. +1 unit 1m 8s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79. -1 asflicense 0m 24s Patch generated 56 ASF License warnings. 201m 59s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSPermission   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.qjournal.client.TestQuorumJournalManager Subsystem Report/Notes Docker Image:yetus/hadoop:date2015-11-20 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773455/HDFS-9038-003.patch JIRA Issue HDFS-9038 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 2ccc37d6f170 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh git revision trunk / 4539131 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt javac hadoop-hdfs-project-jdk1.7.0_79: https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_79.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13575/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13575/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 78MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13575/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Please check the failure of hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics

          Show
          vinayrpet Vinayakumar B added a comment - Please check the failure of hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          This is not failing locally.But there is chance that this testcase can fail intermittently after this change(There is a chance that nonDFS usage might have slightly due to testlogs,).So I am keeping safeside as I did TestNamenodeCapacityReport.java

          Check following results for same

          Testenv dfsused dfsremaining nondfsused TotalCapacity difference
          Eclipse 888 570023337984 74221730952 644245069824 0
          jenkins Run 147456 20430970142720 2887306715136 23318276997120 8192
          Show
          brahmareddy Brahma Reddy Battula added a comment - This is not failing locally.But there is chance that this testcase can fail intermittently after this change(There is a chance that nonDFS usage might have slightly due to testlogs,).So I am keeping safeside as I did TestNamenodeCapacityReport.java Check following results for same Testenv dfsused dfsremaining nondfsused TotalCapacity difference Eclipse 888 570023337984 74221730952 644245069824 0 jenkins Run 147456 20430970142720 2887306715136 23318276997120 8192
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          Hi Brahma Reddy Battula, thanks for taking this up. The approach looks good.

          1. The calculation in FsVolumeImpl#getNonDfsUsed looks wrong. Won't it always be zero? The correct calculation should replace getAvailable() with actual volume free space. Something like Files.getFileStore(currentDir.toPath()).getUnallocatedSpace() perhaps. Also I think reservedForReplicas need not be subtracted in the nonDFSUsed calculation, since it is purely a bytes on disk calculation.
              public long getAvailable() throws IOException {
                long remaining = getCapacity() - getDfsUsed() - reservedForReplicas.get();
            ...
              public long getNonDfsUsed() throws IOException {
                long nonDFSUsed = getCapacity() - getDfsUsed() - getAvailable()
                    - reservedForReplicas.get();
            
          2. Can you add a targeted unit test for the above calculation? A mock-based test would be ideal.
          3. If the DN does not report nonDfsUsed, the NN should initialize it using the older calculation. Else the NN will report nonDfsUsed as zero for older DNs e.g. during upgrade. The fix can be done in PBHelperClient#convert(DatanodeInfoProto).
          4. We can expose nonDfsUsed via StorageTypeStats and DatanodeStatistics. Okay to do in a separate Jira.
          5. Also we can expose reservedForReplicas in a separate Jira. Previously this space was accounted for in nonDfsUsed but now it will not be accounted for anywhere.
          6. Unrelated - we should replace DatanodeInfo constructors with a builder pattern in a separate Jira.
          Show
          arpitagarwal Arpit Agarwal added a comment - - edited Hi Brahma Reddy Battula , thanks for taking this up. The approach looks good. The calculation in FsVolumeImpl#getNonDfsUsed looks wrong. Won't it always be zero? The correct calculation should replace getAvailable() with actual volume free space. Something like Files.getFileStore(currentDir.toPath()).getUnallocatedSpace() perhaps. Also I think reservedForReplicas need not be subtracted in the nonDFSUsed calculation, since it is purely a bytes on disk calculation. public long getAvailable() throws IOException { long remaining = getCapacity() - getDfsUsed() - reservedForReplicas.get(); ... public long getNonDfsUsed() throws IOException { long nonDFSUsed = getCapacity() - getDfsUsed() - getAvailable() - reservedForReplicas.get(); Can you add a targeted unit test for the above calculation? A mock-based test would be ideal. If the DN does not report nonDfsUsed, the NN should initialize it using the older calculation. Else the NN will report nonDfsUsed as zero for older DNs e.g. during upgrade. The fix can be done in PBHelperClient#convert(DatanodeInfoProto) . We can expose nonDfsUsed via StorageTypeStats and DatanodeStatistics. Okay to do in a separate Jira. Also we can expose reservedForReplicas in a separate Jira. Previously this space was accounted for in nonDfsUsed but now it will not be accounted for anywhere. Unrelated - we should replace DatanodeInfo constructors with a builder pattern in a separate Jira.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
          +1 mvninstall 10m 28s trunk passed
          +1 compile 2m 34s trunk passed with JDK v1.8.0_66
          +1 compile 2m 10s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 2m 2s trunk passed
          +1 mvneclipse 0m 35s trunk passed
          +1 findbugs 5m 4s trunk passed
          +1 javadoc 2m 9s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 4s trunk passed with JDK v1.7.0_85
          -1 mvninstall 0m 37s hadoop-hdfs in the patch failed.
          +1 compile 2m 27s the patch passed with JDK v1.8.0_66
          +1 cc 2m 27s the patch passed
          -1 javac 11m 28s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 49).
          +1 javac 2m 27s the patch passed
          +1 compile 2m 0s the patch passed with JDK v1.7.0_85
          +1 cc 2m 0s the patch passed
          -1 javac 13m 28s hadoop-hdfs-project-jdk1.7.0_85 with JDK v1.7.0_85 generated 3 new issues (was 51, now 51).
          +1 javac 2m 0s the patch passed
          -1 checkstyle 0m 28s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 292, now 294).
          +1 mvnsite 1m 55s the patch passed
          +1 mvneclipse 0m 31s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 5m 28s the patch passed
          +1 javadoc 2m 9s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 4s the patch passed with JDK v1.7.0_85
          -1 unit 83m 24s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 1m 30s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 76m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
          +1 unit 1m 22s hadoop-hdfs-client in the patch passed with JDK v1.7.0_85.
          -1 asflicense 0m 25s Patch generated 57 ASF License warnings.
          215m 14s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestSafeModeWithStripedFile
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.qjournal.client.TestQJMWithFaults
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
            hadoop.hdfs.TestRecoverStripedFile
          JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
            hadoop.hdfs.server.namenode.TestBackupNode



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:date2015-11-20
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773567/HDFS-9038-004.patch
          JIRA Issue HDFS-9038
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 060164f9f2ad 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh
          git revision trunk / 4539131
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
          javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt
          javac hadoop-hdfs-project-jdk1.7.0_85: https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_85.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13581/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
          Max memory used 79MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13581/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. +1 mvninstall 10m 28s trunk passed +1 compile 2m 34s trunk passed with JDK v1.8.0_66 +1 compile 2m 10s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 30s trunk passed +1 mvnsite 2m 2s trunk passed +1 mvneclipse 0m 35s trunk passed +1 findbugs 5m 4s trunk passed +1 javadoc 2m 9s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 4s trunk passed with JDK v1.7.0_85 -1 mvninstall 0m 37s hadoop-hdfs in the patch failed. +1 compile 2m 27s the patch passed with JDK v1.8.0_66 +1 cc 2m 27s the patch passed -1 javac 11m 28s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 49). +1 javac 2m 27s the patch passed +1 compile 2m 0s the patch passed with JDK v1.7.0_85 +1 cc 2m 0s the patch passed -1 javac 13m 28s hadoop-hdfs-project-jdk1.7.0_85 with JDK v1.7.0_85 generated 3 new issues (was 51, now 51). +1 javac 2m 0s the patch passed -1 checkstyle 0m 28s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 292, now 294). +1 mvnsite 1m 55s the patch passed +1 mvneclipse 0m 31s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 28s the patch passed +1 javadoc 2m 9s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 4s the patch passed with JDK v1.7.0_85 -1 unit 83m 24s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 1m 30s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 76m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_85. +1 unit 1m 22s hadoop-hdfs-client in the patch passed with JDK v1.7.0_85. -1 asflicense 0m 25s Patch generated 57 ASF License warnings. 215m 14s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.qjournal.client.TestQJMWithFaults   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   hadoop.hdfs.TestRecoverStripedFile JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.server.namenode.TestBackupNode Subsystem Report/Notes Docker Image:yetus/hadoop:date2015-11-20 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773567/HDFS-9038-004.patch JIRA Issue HDFS-9038 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 060164f9f2ad 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh git revision trunk / 4539131 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt javac hadoop-hdfs-project-jdk1.7.0_85: https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_85.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13581/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13581/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 79MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13581/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          The calculation in FsVolumeImpl#getNonDfsUsed looks wrong. Won't it always be zero?

          Do you mean, nonDfsUsed will be zero? from the logs and tests, nonDFSUsage, is giving not zero.
          From the earlier table, Brahma posted, Jenkins machine had 2887306715136 bytes of non-dfs usage out of 23318276997120 capacity, which seems reasonable.

          The correct calculation should replace getAvailable() with actual volume free space. Something like Files.getFileStore(currentDir.toPath()).getUnallocatedSpace() perhaps.

          IMO, This is also good point. Though earlier calculation was not entirely wrong, which basically calculates the nonDfsUsed by subtracting available space and dfsUsed from capacity. Available space is based on File.getUsableSpace() api, which according to javadoc, usable by this JVM not of entire partition.
          Even though, I dont have any objection to use, Files.getFileStore(currentDir.toPath()).getUnallocatedSpace(), how about using currentDir.getFreeSpace()? Do you find any difference between them?
          Updated calculation might look like this

          public long getNonDfsUsed() throws IOException {
              long totalFreeSpace = currentDir.getFreeSpace();
              return getCapacity() - getDfsUsed() - totalFreeSpace;
            }

          Is this looks okay Arpit Agarwal ?

          Show
          vinayrpet Vinayakumar B added a comment - The calculation in FsVolumeImpl#getNonDfsUsed looks wrong. Won't it always be zero? Do you mean, nonDfsUsed will be zero? from the logs and tests, nonDFSUsage, is giving not zero. From the earlier table, Brahma posted, Jenkins machine had 2887306715136 bytes of non-dfs usage out of 23318276997120 capacity, which seems reasonable. The correct calculation should replace getAvailable() with actual volume free space. Something like Files.getFileStore(currentDir.toPath()).getUnallocatedSpace() perhaps. IMO, This is also good point. Though earlier calculation was not entirely wrong, which basically calculates the nonDfsUsed by subtracting available space and dfsUsed from capacity. Available space is based on File.getUsableSpace() api, which according to javadoc, usable by this JVM not of entire partition. Even though, I dont have any objection to use, Files.getFileStore(currentDir.toPath()).getUnallocatedSpace() , how about using currentDir.getFreeSpace() ? Do you find any difference between them? Updated calculation might look like this public long getNonDfsUsed() throws IOException { long totalFreeSpace = currentDir.getFreeSpace(); return getCapacity() - getDfsUsed() - totalFreeSpace; } Is this looks okay Arpit Agarwal ?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Vinayakumar B, yes looks like it will be zero unless the remaining > available check is triggered in #getAvailable.

          Even though, I dont have any objection to use, Files.getFileStore(currentDir.toPath()).getUnallocatedSpace(), how about using currentDir.getFreeSpace()? Do you find any difference between them?

          That makes sense. Your code snippet LGTM.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Vinayakumar B , yes looks like it will be zero unless the remaining > available check is triggered in #getAvailable . Even though, I dont have any objection to use, Files.getFileStore(currentDir.toPath()).getUnallocatedSpace(), how about using currentDir.getFreeSpace()? Do you find any difference between them? That makes sense. Your code snippet LGTM.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks you guys.. soon will update the patch based your comments..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks you guys.. soon will update the patch based your comments..
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Uploaded the patch..Kindly review..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Uploaded the patch..Kindly review..
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Raised separate jira's for above three improvements (HDFS-9480,HDFS-9481 and HDFS-9482).And uploaded the patch to address Vinayakumar B and Arpit Agarwal comments,kindly review..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Raised separate jira's for above three improvements ( HDFS-9480 , HDFS-9481 and HDFS-9482 ).And uploaded the patch to address Vinayakumar B and Arpit Agarwal comments,kindly review..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          +1 mvninstall 14m 23s trunk passed
          +1 compile 3m 20s trunk passed with JDK v1.8.0_66
          +1 compile 2m 33s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 2m 18s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 6m 0s trunk passed
          +1 javadoc 2m 53s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 53s trunk passed with JDK v1.7.0_85
          -1 mvninstall 0m 44s hadoop-hdfs in the patch failed.
          +1 compile 3m 48s the patch passed with JDK v1.8.0_66
          +1 cc 3m 48s the patch passed
          -1 javac 15m 16s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 49).
          +1 javac 3m 48s the patch passed
          +1 compile 2m 36s the patch passed with JDK v1.7.0_85
          +1 cc 2m 37s the patch passed
          -1 javac 17m 54s hadoop-hdfs-project-jdk1.7.0_85 with JDK v1.7.0_85 generated 3 new issues (was 51, now 51).
          +1 javac 2m 36s the patch passed
          -1 checkstyle 0m 35s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 293, now 295).
          +1 mvnsite 2m 48s the patch passed
          +1 mvneclipse 0m 49s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 7m 38s the patch passed
          +1 javadoc 3m 12s the patch passed with JDK v1.8.0_66
          +1 javadoc 4m 49s the patch passed with JDK v1.7.0_85
          -1 unit 115m 25s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 1m 48s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 121m 39s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
          +1 unit 1m 43s hadoop-hdfs-client in the patch passed with JDK v1.7.0_85.
          -1 asflicense 0m 32s Patch generated 57 ASF License warnings.
          311m 11s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestBlockStoragePolicy
            hadoop.tracing.TestTracing
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.TestSafeModeWithStripedFile
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.TestDFSStripedInputStream
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.namenode.snapshot.TestSnapshot
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.blockmanagement.TestPendingReplication
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_85 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.ha.TestLossyRetryInvocationHandler



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774823/HDFS-9038-005.patch
          JIRA Issue HDFS-9038
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 51296cf527bc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c37c3f4
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
          javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt
          javac hadoop-hdfs-project-jdk1.7.0_85: https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_85.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13685/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13685/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. +1 mvninstall 14m 23s trunk passed +1 compile 3m 20s trunk passed with JDK v1.8.0_66 +1 compile 2m 33s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 35s trunk passed +1 mvnsite 2m 18s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 6m 0s trunk passed +1 javadoc 2m 53s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 53s trunk passed with JDK v1.7.0_85 -1 mvninstall 0m 44s hadoop-hdfs in the patch failed. +1 compile 3m 48s the patch passed with JDK v1.8.0_66 +1 cc 3m 48s the patch passed -1 javac 15m 16s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 49). +1 javac 3m 48s the patch passed +1 compile 2m 36s the patch passed with JDK v1.7.0_85 +1 cc 2m 37s the patch passed -1 javac 17m 54s hadoop-hdfs-project-jdk1.7.0_85 with JDK v1.7.0_85 generated 3 new issues (was 51, now 51). +1 javac 2m 36s the patch passed -1 checkstyle 0m 35s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 293, now 295). +1 mvnsite 2m 48s the patch passed +1 mvneclipse 0m 49s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 7m 38s the patch passed +1 javadoc 3m 12s the patch passed with JDK v1.8.0_66 +1 javadoc 4m 49s the patch passed with JDK v1.7.0_85 -1 unit 115m 25s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 1m 48s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 121m 39s hadoop-hdfs in the patch failed with JDK v1.7.0_85. +1 unit 1m 43s hadoop-hdfs-client in the patch passed with JDK v1.7.0_85. -1 asflicense 0m 32s Patch generated 57 ASF License warnings. 311m 11s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestBlockStoragePolicy   hadoop.tracing.TestTracing   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestDFSStripedInputStream   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.blockmanagement.TestPendingReplication   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_85 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.ha.TestLossyRetryInvocationHandler Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774823/HDFS-9038-005.patch JIRA Issue HDFS-9038 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 51296cf527bc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c37c3f4 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt javac hadoop-hdfs-project-jdk1.7.0_85: https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_85.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13685/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13685/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13685/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Thanks for the further reviews. I'm catching up on patch v005 now.

          1. The latest getNonDfsUsed switched to using File#getFreeSpace. However, the getAvailable calculation uses File#getUsableSpace via DF#getAvailable. The non-DFS used calculation prior to the HDFS-5215 patch also would have been using File#getUsableSpace. I think we should stick with File#getUsableSpace here (or DF#getAvailable for symmetry with the pre-HDFS-5215 code).
          2. The latest getNonDfsUsed does not include reservedForReplicas. I think it should, since the reservedForReplicas amount is effectively in use by HDFS.
          3. I think we should cap the returned value to 0 as a matter of defensive coding against negative values. There could be a possibility of race conditions in between pulling the individual data items, resulting in an unexpected negative total.
          Show
          cnauroth Chris Nauroth added a comment - Thanks for the further reviews. I'm catching up on patch v005 now. The latest getNonDfsUsed switched to using File#getFreeSpace . However, the getAvailable calculation uses File#getUsableSpace via DF#getAvailable . The non-DFS used calculation prior to the HDFS-5215 patch also would have been using File#getUsableSpace . I think we should stick with File#getUsableSpace here (or DF#getAvailable for symmetry with the pre- HDFS-5215 code). The latest getNonDfsUsed does not include reservedForReplicas . I think it should, since the reservedForReplicas amount is effectively in use by HDFS. I think we should cap the returned value to 0 as a matter of defensive coding against negative values. There could be a possibility of race conditions in between pulling the individual data items, resulting in an unexpected negative total.
          Hide
          vinayrpet Vinayakumar B added a comment -

          The latest getNonDfsUsed switched to using File#getFreeSpace. However, the getAvailable calculation uses File#getUsableSpace via DF#getAvailable. The non-DFS used calculation prior to the HDFS-5215 patch also would have been using File#getUsableSpace. I think we should stick with File#getUsableSpace here (or DF#getAvailable for symmetry with the pre-HDFS-5215 code).

          I agree that, to be in symmetry with pre HDFS-5215 code, we can use File#getUsableSpace, But since we are trying to calculate entire non-dfs usage of the partition/disk, File#getFreeSpace is required. Both might result in same values, unless some quota set for the user.

          Hi Arpit Agarwal, what you think?

          Show
          vinayrpet Vinayakumar B added a comment - The latest getNonDfsUsed switched to using File#getFreeSpace. However, the getAvailable calculation uses File#getUsableSpace via DF#getAvailable. The non-DFS used calculation prior to the HDFS-5215 patch also would have been using File#getUsableSpace. I think we should stick with File#getUsableSpace here (or DF#getAvailable for symmetry with the pre- HDFS-5215 code). I agree that, to be in symmetry with pre HDFS-5215 code, we can use File#getUsableSpace , But since we are trying to calculate entire non-dfs usage of the partition/disk, File#getFreeSpace is required. Both might result in same values, unless some quota set for the user. Hi Arpit Agarwal , what you think?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the updated patch Brahma Reddy Battula.

          3. I think we should cap the returned value to 0 as a matter of defensive coding against negative values.

          Hi Chris Nauroth, this is a good idea.

          2. The latest getNonDfsUsed does not include reservedForReplicas. I think it should, since the reservedForReplicas amount is effectively in use by HDFS.

          Brahma changed this at my suggestion. Exposing reserved bytes through nonDfsUsed was inadvertent side effect of HDFS-6898. I see Brahma filed HDFS-9481 to expose reserved bytes as a separate counter which should present a clearer picture to administrators. What do you think?

          I think we should stick with File#getUsableSpace here

          I looked at the JDK code to find the difference between File#getUsableSpace and File#getFreeSpace.
          http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/src/solaris/native/java/io/UnixFileSystem_md.c#l412

          So it comes down to the difference between f_bavail and f_bfree.

                     struct statvfs {
                         unsigned long  f_bsize;    /* filesystem block size */
                         unsigned long  f_frsize;   /* fragment size */
                         fsblkcnt_t     f_blocks;   /* size of fs in f_frsize units */
                         fsblkcnt_t     f_bfree;    /* # free blocks */
                         fsblkcnt_t     f_bavail;   /* # free blocks for unprivileged users */
          

          f_bavail does not include some system-reserved partition space. The consequence of using f_bavail is that non-DFS used will include this space which is 5% of the volume size on ext2fs and successors, because File#getTotalSpace counts everything.
          http://www.microhowto.info/howto/reduce_the_space_reserved_for_root_on_an_ext2_ext3_or_ext4_filesystem.html

          This may be confusing to administrators.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the updated patch Brahma Reddy Battula . 3. I think we should cap the returned value to 0 as a matter of defensive coding against negative values. Hi Chris Nauroth , this is a good idea. 2. The latest getNonDfsUsed does not include reservedForReplicas. I think it should, since the reservedForReplicas amount is effectively in use by HDFS. Brahma changed this at my suggestion. Exposing reserved bytes through nonDfsUsed was inadvertent side effect of HDFS-6898 . I see Brahma filed HDFS-9481 to expose reserved bytes as a separate counter which should present a clearer picture to administrators. What do you think? I think we should stick with File#getUsableSpace here I looked at the JDK code to find the difference between File#getUsableSpace and File#getFreeSpace . http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/src/solaris/native/java/io/UnixFileSystem_md.c#l412 So it comes down to the difference between f_bavail and f_bfree . struct statvfs { unsigned long f_bsize; /* filesystem block size */ unsigned long f_frsize; /* fragment size */ fsblkcnt_t f_blocks; /* size of fs in f_frsize units */ fsblkcnt_t f_bfree; /* # free blocks */ fsblkcnt_t f_bavail; /* # free blocks for unprivileged users */ f_bavail does not include some system-reserved partition space. The consequence of using f_bavail is that non-DFS used will include this space which is 5% of the volume size on ext2fs and successors, because File#getTotalSpace counts everything. http://www.microhowto.info/howto/reduce_the_space_reserved_for_root_on_an_ext2_ext3_or_ext4_filesystem.html This may be confusing to administrators.
          Hide
          cnauroth Chris Nauroth added a comment -

          Brahma changed this at my suggestion. Exposing reserved bytes through nonDfsUsed was inadvertent side effect of HDFS-6898.

          Sorry, I missed that earlier comment. I understand the rationale now, so I'll withdraw my comment about changing it.

          Regarding File#getUsableSpace vs. File#getFreeSpace, I've been approaching this as a regression introduced by HDFS-5215, with the goal of restoring the pre-HDFS-5215 behavior as closely as possible. Before HDFS-5215, non-DFS used was based on File#getUsableSpace, indirectly because of use of getAvailable in the calculation. This meant that "free but unusable" space would have counted towards non-DFS used.

          However, that would then lead me to expect seeing a non-zero non-DFS used for the typical 5% reserved for privileged users. In practice, I haven't seen that happen though. I'm not sure why not.

          I tried digging through revision history for past decisions. The use of File#getUsableSpace traces back to HADOOP-5958, when the code switched from forking to df to using then-new JDK 1.6 APIs. There is a question posted about which method to use, but no specific discussion about why File#getUsableSpace was chosen.

          I don't think switching to File#getFreeSpace is necessarily wrong, just different from the pre-HDFS-5215 calculation, which might cause some surprises.

          Show
          cnauroth Chris Nauroth added a comment - Brahma changed this at my suggestion. Exposing reserved bytes through nonDfsUsed was inadvertent side effect of HDFS-6898 . Sorry, I missed that earlier comment. I understand the rationale now, so I'll withdraw my comment about changing it. Regarding File#getUsableSpace vs. File#getFreeSpace , I've been approaching this as a regression introduced by HDFS-5215 , with the goal of restoring the pre- HDFS-5215 behavior as closely as possible. Before HDFS-5215 , non-DFS used was based on File#getUsableSpace , indirectly because of use of getAvailable in the calculation. This meant that "free but unusable" space would have counted towards non-DFS used. However, that would then lead me to expect seeing a non-zero non-DFS used for the typical 5% reserved for privileged users. In practice, I haven't seen that happen though. I'm not sure why not. I tried digging through revision history for past decisions. The use of File#getUsableSpace traces back to HADOOP-5958 , when the code switched from forking to df to using then-new JDK 1.6 APIs. There is a question posted about which method to use, but no specific discussion about why File#getUsableSpace was chosen. I don't think switching to File#getFreeSpace is necessarily wrong, just different from the pre- HDFS-5215 calculation, which might cause some surprises.
          Hide
          aw Allen Wittenauer added a comment -

          I wonder how these various methods react with pooled storage....

          Show
          aw Allen Wittenauer added a comment - I wonder how these various methods react with pooled storage....
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          I don't think switching to File#getFreeSpace is necessarily wrong, just different from the pre-HDFS-5215 calculation, which might cause some surprises.

          Yes, its surprising.
          Difference in HDFS user about that ~5% space,
          before patch --> It was taken as already used by someone (non-dfs) because its not usable by HDFS.
          after patch --> It will display actual usage of someone (non-dfs) even though its not usable.

          Only matter is, value of metric. Which doesnt affect HDFS functionality anyway.
          But assumption in calculattion in tests capacityUsed + capacityRemaining + capacityUsedNonDFS == capacityTotal might need to change, as capacityRemaining doesnt include, that ~5%.

          From the above Test Failure of TestNameNodeMetrics, below values have got

          CapacityTotal 23318276997120
          CapacityUsed 147456
          CapacityRemaining 20273827930112
          CapacityUsedNonDFS 1859806781440
          Difference ~5%

          Now we have two options to go ahead with the patch.

          • Update the tests with assumption of extra ~5-6% space for privileged users
          • fallback to File.getUsablespace()

          Which one do you prefer Chris Nauroth,Arpit Agarwal..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - I don't think switching to File#getFreeSpace is necessarily wrong, just different from the pre- HDFS-5215 calculation, which might cause some surprises. Yes, its surprising. Difference in HDFS user about that ~5% space, before patch --> It was taken as already used by someone (non-dfs) because its not usable by HDFS. after patch --> It will display actual usage of someone (non-dfs) even though its not usable. Only matter is, value of metric. Which doesnt affect HDFS functionality anyway. But assumption in calculattion in tests capacityUsed + capacityRemaining + capacityUsedNonDFS == capacityTotal might need to change, as capacityRemaining doesnt include, that ~5% . From the above Test Failure of TestNameNodeMetrics , below values have got CapacityTotal 23318276997120 CapacityUsed 147456 CapacityRemaining 20273827930112 CapacityUsedNonDFS 1859806781440 Difference ~5% Now we have two options to go ahead with the patch. Update the tests with assumption of extra ~5-6% space for privileged users fallback to File.getUsablespace() Which one do you prefer Chris Nauroth , Arpit Agarwal ..?
          Hide
          vinayrpet Vinayakumar B added a comment -

          btw, Allen Wittenauer, is YETUS-170 included in recent precommit builds? when I checked for above failure, it was not there, so mvn install failed due to wrong order.

          Show
          vinayrpet Vinayakumar B added a comment - btw, Allen Wittenauer , is YETUS-170 included in recent precommit builds? when I checked for above failure, it was not there, so mvn install failed due to wrong order.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -
          Show
          brahmareddy Brahma Reddy Battula added a comment - Ping Chris Nauroth / Arpit Agarwal / Vinayakumar B once again..
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm in favor of using File#getUsableSpace for the sake of consistency with the pre-HDFS-5215 behavior. I'd also like to make sure Arpit gets a chance to respond too, but I don't expect he'll have a chance to comment until next week.

          Show
          cnauroth Chris Nauroth added a comment - I'm in favor of using File#getUsableSpace for the sake of consistency with the pre- HDFS-5215 behavior. I'd also like to make sure Arpit gets a chance to respond too, but I don't expect he'll have a chance to comment until next week.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Ok.will wait till Arpit Agarwal give his thought.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Ok.will wait till Arpit Agarwal give his thought.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          Sorry for the delay in responding, I am out this week. I just did some quick testing with a single DN using a 40GB ext4 partition.

          With Brahma's v005 patch modified to use File#getUsableSpace non-DFS usage is reported as 2GB, i.e. ~5%. The actual disk usage was 49MB.

          $ bin/hdfs dfsadmin -report
          ...
          Live datanodes (1):
          
          Name: 127.0.0.1:50010 (localhost)
          Hostname: mint0
          Decommission Status : Normal
          Configured Capacity: 42141548544 (39.25 GB)
          DFS Used: 24576 (24 KB)
          Non DFS Used: 2215555072 (2.06 GB)  <<<<
          DFS Remaining: 39925968896 (37.18 GB)
          DFS Used%: 0.00%
          DFS Remaining%: 94.74%
          
          $ df -h /mnt/sdb/
          Filesystem      Size  Used Avail Use% Mounted on
          /dev/sdb         40G   49M   38G   1% /mnt/sdb <<<<
          

          With File#getFreeSpace i.e. the v005 patch, the non-DFS used is more accurate.

          $ bin/hdfs dfsadmin -report
          
          ...
          Live datanodes (1):
          
          Name: 127.0.0.1:50010 (localhost)
          Hostname: mint0
          Decommission Status : Normal
          Configured Capacity: 42141548544 (39.25 GB)
          DFS Used: 24576 (24 KB)
          Non DFS Used: 51302400 (48.93 MB)
          

          I also checked the current behavior in trunk (no patch). It is already broken, counting the system-reserved space towards non-DFS used.

          $ bin/hdfs dfsadmin -report
          ...
          
          Name: 127.0.0.1:50010 (localhost)
          Hostname: mint0
          Decommission Status : Normal
          Configured Capacity: 42141548544 (39.25 GB)
          DFS Used: 24576 (24 KB)
          Non DFS Used: 2215555072 (2.06 GB)
          

          I think File#getFreeSpace is the correct choice. However modifying Brahma's patch use File#getUsableSpace would not introduce a regression wrt what's in trunk already so it's fine to change it to address Chris's concern. I'll file a separate Jira to discuss fixing this part.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited Sorry for the delay in responding, I am out this week. I just did some quick testing with a single DN using a 40GB ext4 partition. With Brahma's v005 patch modified to use File#getUsableSpace non-DFS usage is reported as 2GB, i.e. ~5%. The actual disk usage was 49MB. $ bin/hdfs dfsadmin -report ... Live datanodes (1): Name: 127.0.0.1:50010 (localhost) Hostname: mint0 Decommission Status : Normal Configured Capacity: 42141548544 (39.25 GB) DFS Used: 24576 (24 KB) Non DFS Used: 2215555072 (2.06 GB) <<<< DFS Remaining: 39925968896 (37.18 GB) DFS Used%: 0.00% DFS Remaining%: 94.74% $ df -h /mnt/sdb/ Filesystem Size Used Avail Use% Mounted on /dev/sdb 40G 49M 38G 1% /mnt/sdb <<<< With File#getFreeSpace i.e. the v005 patch, the non-DFS used is more accurate. $ bin/hdfs dfsadmin -report ... Live datanodes (1): Name: 127.0.0.1:50010 (localhost) Hostname: mint0 Decommission Status : Normal Configured Capacity: 42141548544 (39.25 GB) DFS Used: 24576 (24 KB) Non DFS Used: 51302400 (48.93 MB) I also checked the current behavior in trunk (no patch). It is already broken, counting the system-reserved space towards non-DFS used. $ bin/hdfs dfsadmin -report ... Name: 127.0.0.1:50010 (localhost) Hostname: mint0 Decommission Status : Normal Configured Capacity: 42141548544 (39.25 GB) DFS Used: 24576 (24 KB) Non DFS Used: 2215555072 (2.06 GB) I think File#getFreeSpace is the correct choice. However modifying Brahma's patch use File#getUsableSpace would not introduce a regression wrt what's in trunk already so it's fine to change it to address Chris's concern. I'll file a separate Jira to discuss fixing this part.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          I also edited the Jira title to mention "DFS reserved space", to distinguish it from system-reserved space.

          Show
          arpitagarwal Arpit Agarwal added a comment - I also edited the Jira title to mention "DFS reserved space", to distinguish it from system-reserved space.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Arpit Agarwal updated previous patch(004) as per Chris comments and added check for -ve values..

          I'll file a separate Jira to discuss fixing this part.

          will discuss further in seperate jira.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Arpit Agarwal updated previous patch(004) as per Chris comments and added check for -ve values.. I'll file a separate Jira to discuss fixing this part. will discuss further in seperate jira.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          +1 mvninstall 8m 6s trunk passed
          +1 compile 1m 45s trunk passed with JDK v1.8.0_66
          +1 compile 1m 45s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 1m 31s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 4m 5s trunk passed
          +1 javadoc 1m 33s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 18s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 29s the patch passed
          +1 compile 1m 53s the patch passed with JDK v1.8.0_66
          +1 cc 1m 53s the patch passed
          -1 javac 9m 16s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 49).
          +1 javac 1m 53s the patch passed
          +1 compile 1m 50s the patch passed with JDK v1.7.0_91
          +1 cc 1m 50s the patch passed
          -1 javac 11m 6s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 51, now 51).
          +1 javac 1m 50s the patch passed
          -1 checkstyle 0m 23s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 292, now 294).
          +1 mvnsite 1m 39s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 4m 31s the patch passed
          +1 javadoc 1m 41s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 27s the patch passed with JDK v1.7.0_91
          +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 67m 17s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 1m 7s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          -1 unit 66m 31s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 19s Patch generated 58 ASF License warnings.
          177m 30s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestFileCreationDelete
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList
            hadoop.hdfs.server.namenode.TestFsck
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.datanode.TestFsDatasetCache
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.server.namenode.TestRecoverStripedBlocks
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList
            hadoop.hdfs.server.namenode.TestCacheDirectives



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776866/HDFS-9038-006.patch
          JIRA Issue HDFS-9038
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 2d8f7c9f185e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 21daa6c
          findbugs v3.0.0
          javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt
          javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13828/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
          Max memory used 76MB
          Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13828/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. +1 mvninstall 8m 6s trunk passed +1 compile 1m 45s trunk passed with JDK v1.8.0_66 +1 compile 1m 45s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 31s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 4m 5s trunk passed +1 javadoc 1m 33s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 18s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 29s the patch passed +1 compile 1m 53s the patch passed with JDK v1.8.0_66 +1 cc 1m 53s the patch passed -1 javac 9m 16s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 49). +1 javac 1m 53s the patch passed +1 compile 1m 50s the patch passed with JDK v1.7.0_91 +1 cc 1m 50s the patch passed -1 javac 11m 6s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 51, now 51). +1 javac 1m 50s the patch passed -1 checkstyle 0m 23s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 292, now 294). +1 mvnsite 1m 39s the patch passed +1 mvneclipse 0m 27s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 4m 31s the patch passed +1 javadoc 1m 41s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 27s the patch passed with JDK v1.7.0_91 +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 67m 17s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 1m 7s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 66m 31s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 19s Patch generated 58 ASF License warnings. 177m 30s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestFileCreationDelete   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.datanode.TestFsDatasetCache JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.server.namenode.TestRecoverStripedBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList   hadoop.hdfs.server.namenode.TestCacheDirectives Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776866/HDFS-9038-006.patch JIRA Issue HDFS-9038 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 2d8f7c9f185e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 21daa6c findbugs v3.0.0 javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13828/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13828/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 76MB Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13828/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I took patch v006 for a more thorough test run. Unfortunately, I'm still seeing the bug that was introduced by HDFS-5215. Even though non-DFS used is now calculated explicitly by the DataNode and sent in a dedicated RPC field, it's still using an incorrect calculation.

          To help illustrate the problem better, I'd like to walk through the math for calculating non-DFS used, both before HDFS-5215 and after HDFS-5215. Here is a bit of pseudo-code, prefixed with line numbers.

          Before HDFS-5215:

          1: nonDfsUsed = capacity - dfsUsed - available
          2:            = (File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - available
          3:            = (File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - File#getUsableSpace
          4:            = File#getTotalSpace - dfs.datanode.du.reserved - dfsUsed - File#getUsableSpace
          

          After HDFS-5215

          1: nonDfsUsed = capacity - dfsUsed - available
          2:            = (File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - available
          3:            = (File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - (File#getUsableSpace - dfs.datanode.du.reserved)
          4:            = File#getTotalSpace - dfs.datanode.du.reserved - dfsUsed - File#getUsableSpace + dfs.datanode.du.reserved
          5:            = File#getTotalSpace - dfsUsed - File#getUsableSpace
          

          The most important point to note is the difference in line 3 of the expansions. After HDFS-5215, the definition of available removes dfs.datanode.du.reserved. However, the definition of capacity already removed dfs.datanode.du.reserved. Carrying out the expansion further in lines 4 and 5, we see that the 2 occurrences of dfs.datanode.du.reserved cancel each other out. That means implicitly that nonDfsUsed now includes (instead of excludes) dfs.datanode.du.reserved.

          Patch v006 still has the same fundamental problem in the logic, because FsVolumeImpl#getNonDfsUsed still reuses the implementation of FsVolumeImpl#getAvailable, which in turn removes dfs.datanode.du.reserved.

          I think FsVolumeImpl#getNonDfsUsed essentially needs to go back to the old pre-HDFS-5215 calculation of remaining instead of calling FsVolumeImpl#getAvailable.

          Show
          cnauroth Chris Nauroth added a comment - I took patch v006 for a more thorough test run. Unfortunately, I'm still seeing the bug that was introduced by HDFS-5215 . Even though non-DFS used is now calculated explicitly by the DataNode and sent in a dedicated RPC field, it's still using an incorrect calculation. To help illustrate the problem better, I'd like to walk through the math for calculating non-DFS used, both before HDFS-5215 and after HDFS-5215 . Here is a bit of pseudo-code, prefixed with line numbers. Before HDFS-5215 : 1: nonDfsUsed = capacity - dfsUsed - available 2: = (File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - available 3: = (File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - File#getUsableSpace 4: = File#getTotalSpace - dfs.datanode.du.reserved - dfsUsed - File#getUsableSpace After HDFS-5215 1: nonDfsUsed = capacity - dfsUsed - available 2: = (File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - available 3: = (File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - (File#getUsableSpace - dfs.datanode.du.reserved) 4: = File#getTotalSpace - dfs.datanode.du.reserved - dfsUsed - File#getUsableSpace + dfs.datanode.du.reserved 5: = File#getTotalSpace - dfsUsed - File#getUsableSpace The most important point to note is the difference in line 3 of the expansions. After HDFS-5215 , the definition of available removes dfs.datanode.du.reserved . However, the definition of capacity already removed dfs.datanode.du.reserved . Carrying out the expansion further in lines 4 and 5, we see that the 2 occurrences of dfs.datanode.du.reserved cancel each other out. That means implicitly that nonDfsUsed now includes (instead of excludes) dfs.datanode.du.reserved . Patch v006 still has the same fundamental problem in the logic, because FsVolumeImpl#getNonDfsUsed still reuses the implementation of FsVolumeImpl#getAvailable , which in turn removes dfs.datanode.du.reserved . I think FsVolumeImpl#getNonDfsUsed essentially needs to go back to the old pre- HDFS-5215 calculation of remaining instead of calling FsVolumeImpl#getAvailable .
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks Chris Nauroth for the explanation.

          Lets go through one simple example.
          Lets consider following
          1. Disk Capacity - 10GB – File#getTotalSpace
          2. DFS reserved - 1GB – dfs.datanode.du.reserved
          3. DFS Usage - 2GB – dfsUsed
          4. Usage by other files - 3GB – Actual non-dfsusage.

          Assuming, System reserved ( which is ~5%) as 0 for this example.
          total disk used is 5GB = 2GB by dfs files + 3GB by non-dfs files.
          File.getUsableSpace() returns 5GB.

          Lets substitute this values in derived expressions.
          Before HDFS-5215

          nonDfsUsage=File#getTotalSpace - dfs.datanode.du.reserved - dfsUsed - File#getUsableSpace
          
          nonDfsUsage = 10G - 1G - 2G - 5G
          nonDfsUsage = 2G.
          

          nonDfsUsage=2GB is not same as actual non-dfs usage, 3GB.

          After HDFS-5215 and as per patch.

          nonDfsUsage=(File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - (File#getUsableSpace - dfs.datanode.du.reserved)
          nonDfSUsage= (10G - 1G ) - 2G - (5G - 1G)
          nonDfSUsage = 9G - 2G - 4G
          nonDfsUsage = 3G
          

          nonDfsUsage = 3G is same as expected actual usage. i.e. 3G

          By seeing the above example,
          I believe, post HDFS-5215, calculation turns out to be the correct one.
          dfs.datanode,du.reserved is expected to be the freespace, which should not be used by DFS. So its reasonable to subtract this in getAvailable(), but not while finding the non-dfs usage.

          Do you agree Chris Nauroth?

          Show
          vinayrpet Vinayakumar B added a comment - Thanks Chris Nauroth for the explanation. Lets go through one simple example. Lets consider following 1. Disk Capacity - 10GB – File#getTotalSpace 2. DFS reserved - 1GB – dfs.datanode.du.reserved 3. DFS Usage - 2GB – dfsUsed 4. Usage by other files - 3GB – Actual non-dfsusage . Assuming, System reserved ( which is ~5%) as 0 for this example. total disk used is 5GB = 2GB by dfs files + 3GB by non-dfs files. File.getUsableSpace() returns 5GB. Lets substitute this values in derived expressions. Before HDFS-5215 nonDfsUsage=File#getTotalSpace - dfs.datanode.du.reserved - dfsUsed - File#getUsableSpace nonDfsUsage = 10G - 1G - 2G - 5G nonDfsUsage = 2G. nonDfsUsage=2GB is not same as actual non-dfs usage, 3GB. After HDFS-5215 and as per patch. nonDfsUsage=(File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - (File#getUsableSpace - dfs.datanode.du.reserved) nonDfSUsage= (10G - 1G ) - 2G - (5G - 1G) nonDfSUsage = 9G - 2G - 4G nonDfsUsage = 3G nonDfsUsage = 3G is same as expected actual usage. i.e. 3G By seeing the above example, I believe, post HDFS-5215 , calculation turns out to be the correct one. dfs.datanode,du.reserved is expected to be the freespace, which should not be used by DFS. So its reasonable to subtract this in getAvailable(), but not while finding the non-dfs usage. Do you agree Chris Nauroth ?
          Hide
          vinayrpet Vinayakumar B added a comment -

          Meanwhile, Brahma Reddy Battula, forgot to update the test TestFsVolumeList.testNonDfsUsedMetricForVolume?

          Show
          vinayrpet Vinayakumar B added a comment - Meanwhile, Brahma Reddy Battula , forgot to update the test TestFsVolumeList.testNonDfsUsedMetricForVolume ?
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Vinayakumar B thanks for pointing..Corrected and uploaded the patch.

          Apart from that added null check in TestNameNodeMetrics#tearDown and fixed javadoc error in FsVolumeImpl#getDataset which are not related this jira.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Vinayakumar B thanks for pointing..Corrected and uploaded the patch. Apart from that added null check in TestNameNodeMetrics#tearDown and fixed javadoc error in FsVolumeImpl#getDataset which are not related this jira.
          Hide
          cnauroth Chris Nauroth added a comment -

          I believe, post HDFS-5215, calculation turns out to be the correct one.

          I disagree, because the post-HDFS-5215 calculation has broken previously established operations workflows.

          Non-DFS usage is "space unexpectedly consumed on data volumes by things that are not HDFS". As a cluster administrator, I would monitor the non-DFS usage value. If a node consistently showed a non-zero non-DFS usage, then that would signal me to login to the box, figure out where the space was consumed, free it up, and then address root cause (probably a rogue process running on the wrong box or misconfigured to write to the wrong volume). It's important to fix this, because high non-DFS usage reduces disk capacity that had been planned for HDFS.

          After HDFS-5215, this workflow no longer works. The monitoring will show false positives because of inclusion of dfs.datanode.du.reserved. An administrator would need to use additional checks, or simply disable this monitoring due to the noise.

          dfs.datanode.du.reserved is a special case on top of what I described above. Setting a non-zero dfs.datanode.du.reserved means that the administrator intentionally wants to hold back a portion of the volume, essentially making it invisible to HDFS. This is the same reason that dfs.datanode.du.reserved gets subtracted from the capacity calculation. This is why I think 3 GB is incorrect in the example above. Non-DFS usage is really "unexpected non-DFS usage". By setting dfs.datanode.du.reserved to 1 GB, you have stated that up to 1 GB of usage by something other than HDFS is expected, so it's incorrect to count it as part of "unexpected non-DFS usage".

          Bottom line: I have seen the post-HDFS-5215 calculation cause confusion for administrators who had built a workflow around the old calculation.

          Show
          cnauroth Chris Nauroth added a comment - I believe, post HDFS-5215 , calculation turns out to be the correct one. I disagree, because the post- HDFS-5215 calculation has broken previously established operations workflows. Non-DFS usage is "space unexpectedly consumed on data volumes by things that are not HDFS". As a cluster administrator, I would monitor the non-DFS usage value. If a node consistently showed a non-zero non-DFS usage, then that would signal me to login to the box, figure out where the space was consumed, free it up, and then address root cause (probably a rogue process running on the wrong box or misconfigured to write to the wrong volume). It's important to fix this, because high non-DFS usage reduces disk capacity that had been planned for HDFS. After HDFS-5215 , this workflow no longer works. The monitoring will show false positives because of inclusion of dfs.datanode.du.reserved . An administrator would need to use additional checks, or simply disable this monitoring due to the noise. dfs.datanode.du.reserved is a special case on top of what I described above. Setting a non-zero dfs.datanode.du.reserved means that the administrator intentionally wants to hold back a portion of the volume, essentially making it invisible to HDFS. This is the same reason that dfs.datanode.du.reserved gets subtracted from the capacity calculation. This is why I think 3 GB is incorrect in the example above. Non-DFS usage is really "unexpected non-DFS usage". By setting dfs.datanode.du.reserved to 1 GB, you have stated that up to 1 GB of usage by something other than HDFS is expected, so it's incorrect to count it as part of "unexpected non-DFS usage". Bottom line: I have seen the post- HDFS-5215 calculation cause confusion for administrators who had built a workflow around the old calculation.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          +1 mvninstall 16m 3s trunk passed
          +1 compile 4m 56s trunk passed with JDK v1.8.0_66
          +1 compile 3m 30s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 45s trunk passed
          +1 mvnsite 3m 9s trunk passed
          +1 mvneclipse 0m 51s trunk passed
          +1 findbugs 7m 36s trunk passed
          +1 javadoc 3m 46s trunk passed with JDK v1.8.0_66
          +1 javadoc 4m 42s trunk passed with JDK v1.7.0_91
          +1 mvninstall 3m 0s the patch passed
          +1 compile 5m 0s the patch passed with JDK v1.8.0_66
          +1 cc 5m 0s the patch passed
          -1 javac 20m 19s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 49).
          +1 javac 5m 0s the patch passed
          +1 compile 3m 35s the patch passed with JDK v1.7.0_91
          +1 cc 3m 35s the patch passed
          -1 javac 23m 54s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 51, now 51).
          +1 javac 3m 35s the patch passed
          -1 checkstyle 0m 44s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 293, now 295).
          +1 mvnsite 3m 12s the patch passed
          +1 mvneclipse 0m 50s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 8m 45s the patch passed
          +1 javadoc 3m 54s the patch passed with JDK v1.8.0_66
          +1 javadoc 4m 55s the patch passed with JDK v1.7.0_91
          +1 unit 2m 24s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 105m 50s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 2m 8s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          -1 unit 1m 14s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 36s Patch generated 61 ASF License warnings.
          198m 16s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestParallelShortCircuitRead
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
            hadoop.fs.contract.hdfs.TestHDFSContractMkdir
            hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens
            hadoop.hdfs.TestBlockStoragePolicy
            hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM
            hadoop.cli.TestDeleteCLI
            hadoop.hdfs.tools.TestDFSZKFailoverController
            hadoop.hdfs.TestFileLengthOnClusterRestart
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory
            hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
            hadoop.cli.TestHDFSCLI
            hadoop.hdfs.server.namenode.TestCheckpoint
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.tools.TestGetGroups
            hadoop.hdfs.TestRemoteBlockReader2
            hadoop.hdfs.server.namenode.TestStartup
            hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
            hadoop.hdfs.TestDFSStorageStateRecovery
            hadoop.hdfs.server.namenode.TestFSImageWithXAttr
            hadoop.hdfs.TestRemoteBlockReader
            hadoop.hdfs.TestMultiThreadedHflush
            hadoop.fs.contract.hdfs.TestHDFSContractRename
            hadoop.hdfs.TestBlockReaderLocal
            hadoop.cli.TestCacheAdminCLI
            hadoop.hdfs.server.mover.TestMover
            hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits
            hadoop.hdfs.server.namenode.TestNameNodeRecovery
            hadoop.hdfs.server.namenode.ha.TestFailureOfSharedDir
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.fs.loadGenerator.TestLoadGenerator
            hadoop.hdfs.server.namenode.TestFSImageWithAcl
            hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
            hadoop.fs.TestFcHdfsSetUMask
            hadoop.hdfs.TestPread
            hadoop.hdfs.server.namenode.ha.TestQuotasWithHA
            hadoop.hdfs.crypto.TestHdfsCryptoStreams
            hadoop.fs.viewfs.TestViewFsFileStatusHdfs
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl
            hadoop.hdfs.TestDFSAddressConfig
            hadoop.tracing.TestTracingShortCircuitLocalRead
            hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
            hadoop.hdfs.server.namenode.TestFSDirectory
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.tracing.TestTracing
            hadoop.hdfs.TestParallelRead
            hadoop.hdfs.TestDFSStripedInputStream
            hadoop.hdfs.tools.TestDelegationTokenFetcher
            hadoop.hdfs.TestRestartDFS
            hadoop.fs.TestWebHdfsFileContextMainOperations
            hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
            hadoop.hdfs.TestParallelShortCircuitLegacyRead
            hadoop.net.TestNetworkTopology
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
            hadoop.fs.TestFcHdfsCreateMkdir
            hadoop.hdfs.server.namenode.TestDeadDatanode
            hadoop.hdfs.server.namenode.ha.TestNNHealthCheck
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled
            hadoop.hdfs.TestFetchImage
            hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
            hadoop.hdfs.server.namenode.ha.TestDNFencing
            hadoop.hdfs.TestDFSUpgrade
            hadoop.fs.TestFcHdfsPermission
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks
            hadoop.hdfs.server.namenode.TestProtectedDirectories
            hadoop.hdfs.TestMissingBlocksAlert
            hadoop.hdfs.server.namenode.ha.TestHAMetrics
            hadoop.hdfs.TestQuota
            hadoop.hdfs.server.namenode.TestQuotaByStorageType
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
            hadoop.hdfs.server.namenode.TestSaveNamespace
            hadoop.hdfs.tools.TestDebugAdmin
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover
            hadoop.hdfs.server.namenode.TestNameNodeRpcServer
            hadoop.hdfs.server.namenode.TestSecureNameNode
            hadoop.hdfs.server.namenode.TestSnapshotPathINodes
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
            hadoop.hdfs.TestModTime
            hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA
            hadoop.hdfs.TestGetBlocks
            hadoop.hdfs.TestDFSStartupVersions
            hadoop.hdfs.TestRenameWhileOpen
            hadoop.fs.viewfs.TestViewFsWithXAttrs
            hadoop.cli.TestErasureCodingCLI
            hadoop.tools.TestJMXGet
            hadoop.hdfs.TestFileAppend4
            hadoop.TestGenericRefresh
            hadoop.hdfs.TestRecoverStripedFile
            hadoop.hdfs.server.namenode.TestAddBlock
            hadoop.hdfs.protocol.TestLocatedBlock
            hadoop.hdfs.TestDataTransferKeepalive
            hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
            hadoop.hdfs.TestFileAppend2
            hadoop.hdfs.TestErasureCodeBenchmarkThroughput
            hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.namenode.TestAuditLogs
            hadoop.fs.TestSWebHdfsFileContextMainOperations
            hadoop.hdfs.TestInjectionForSimulatedStorage
            hadoop.hdfs.server.namenode.TestFsck
            hadoop.fs.contract.hdfs.TestHDFSContractDelete
            hadoop.fs.contract.hdfs.TestHDFSContractSetTimes
            hadoop.hdfs.TestLocalDFS
            hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr
            hadoop.hdfs.TestConnCache
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
            hadoop.hdfs.server.namenode.TestStripedINodeFile
            hadoop.hdfs.TestGetFileChecksum
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.TestReservedRawPaths
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.TestRefreshCallQueue
            hadoop.hdfs.TestDisableConnCache
            hadoop.cli.TestXAttrCLI
            hadoop.hdfs.server.namenode.TestMetadataVersionOutput
            hadoop.hdfs.TestParallelShortCircuitReadUnCached
            hadoop.fs.TestUrlStreamHandler
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020
            hadoop.fs.TestEnhancedByteBufferAccess
            hadoop.hdfs.TestHDFSServerPorts
            hadoop.hdfs.server.namenode.TestAuditLogger
            hadoop.hdfs.TestRollingUpgrade
            hadoop.fs.viewfs.TestViewFileSystemWithXAttrs
            hadoop.fs.TestHDFSFileContextMainOperations
            hadoop.hdfs.TestSafeMode
            hadoop.hdfs.protocolPB.TestPBHelper
            hadoop.hdfs.server.namenode.TestBackupNode
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement
            hadoop.hdfs.server.namenode.TestAclConfigFlag
            hadoop.hdfs.TestFileStatusWithECPolicy
            hadoop.hdfs.TestWriteRead
            hadoop.hdfs.tools.TestDFSAdminWithHA
            hadoop.hdfs.TestExternalBlockReader
            hadoop.hdfs.server.namenode.TestMetaSave
            hadoop.fs.contract.hdfs.TestHDFSContractAppend
            hadoop.hdfs.server.namenode.ha.TestStateTransitionFailure
            hadoop.hdfs.TestDFSStripedOutputStream
            hadoop.hdfs.TestHdfsAdmin
            hadoop.fs.contract.hdfs.TestHDFSContractOpen
            hadoop.hdfs.server.namenode.TestAddStripedBlocks
            hadoop.hdfs.server.namenode.TestGenericJournalConf
            hadoop.hdfs.TestFileCreation
            hadoop.hdfs.server.mover.TestStorageMover
            hadoop.hdfs.TestClientReportBadBlock
            hadoop.tracing.TestTraceAdmin
            hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
            hadoop.hdfs.server.namenode.TestBlockUnderConstruction
            hadoop.hdfs.TestFileCorruption
            hadoop.hdfs.server.namenode.TestHDFSConcat
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.TestClose
            hadoop.hdfs.TestAbandonBlock
            hadoop.hdfs.server.namenode.ha.TestHAFsck
            hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant
            hadoop.hdfs.TestDFSInputStream
            hadoop.hdfs.TestLeaseRecovery
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.tools.TestDFSAdmin
            hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
            hadoop.hdfs.TestWriteBlockGetsBlockLengthHint
            hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
            hadoop.fs.viewfs.TestViewFsHdfs
            hadoop.hdfs.TestDFSClientSocketSize
            hadoop.fs.contract.hdfs.TestHDFSContractConcat
            hadoop.hdfs.server.namenode.TestFileLimit
            hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot
            hadoop.hdfs.TestHDFSTrash
            hadoop.hdfs.TestSmallBlock
            hadoop.cli.TestCryptoAdminCLI
            hadoop.hdfs.TestFileCreationDelete
            hadoop.hdfs.server.namenode.ha.TestLossyRetryInvocationHandler
            hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
            hadoop.hdfs.TestReplication
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.cli.TestAclCLI
            hadoop.fs.viewfs.TestViewFileSystemWithAcls
            hadoop.hdfs.server.namenode.TestEditLogAutoroll
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
            hadoop.hdfs.server.namenode.TestTransferFsImage
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.fs.TestSymlinkHdfsFileSystem
            hadoop.hdfs.TestFileAppend3
            hadoop.fs.TestUnbuffer
            hadoop.hdfs.TestPipelines
            hadoop.hdfs.TestErasureCodingPolicies
            hadoop.hdfs.TestAclsEndToEnd
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.namenode.ha.TestHASafeMode
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.TestBlockReaderFactory
            hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.TestFSOutputSummer
            hadoop.hdfs.TestBlockReaderLocalLegacy
            hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
            hadoop.fs.viewfs.TestViewFsWithAcls
            hadoop.hdfs.server.namenode.ha.TestHAStateTransitions
            hadoop.hdfs.TestEncryptionZones
            hadoop.fs.TestResolveHdfsSymlink
            hadoop.hdfs.server.namenode.TestCommitBlockWithInvalidGenStamp
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
            hadoop.hdfs.TestFileConcurrentReader
            hadoop.fs.viewfs.TestViewFsDefaultValue
            hadoop.hdfs.TestAppendDifferentChecksum
            hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus
            hadoop.hdfs.server.namenode.TestCreateEditsLog
            hadoop.hdfs.server.namenode.TestFSNamesystemMBean
            hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA
            hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary
            hadoop.hdfs.server.namenode.TestNameNodeXAttr
            hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
            hadoop.fs.viewfs.TestViewFileSystemHdfs
            hadoop.hdfs.TestParallelUnixDomainRead
            hadoop.hdfs.tools.TestStoragePolicyCommands
            hadoop.hdfs.TestDFSRemove
            hadoop.fs.viewfs.TestViewFsAtHdfsRoot
            hadoop.hdfs.qjournal.TestNNWithQJM
            hadoop.hdfs.TestFileCreationClient
            hadoop.hdfs.TestRollingUpgradeDowngrade
            hadoop.fs.contract.hdfs.TestHDFSContractCreate
            hadoop.fs.permission.TestStickyBit
            hadoop.hdfs.TestEncryptionZonesWithKMS
            hadoop.fs.TestSymlinkHdfsDisable
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.server.namenode.TestDeleteRace
            hadoop.hdfs.TestLease
            hadoop.hdfs.TestLargeBlock
            hadoop.fs.TestGlobPaths
            hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes
            hadoop.hdfs.server.namenode.TestEditLogJournalFailures
            hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys
            hadoop.hdfs.TestClientProtocolForPipelineRecovery
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.namenode.TestSecondaryWebUi
            hadoop.fs.contract.hdfs.TestHDFSContractSeek
            hadoop.fs.shell.TestHdfsTextCommand
            hadoop.hdfs.TestBlockMissingException
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestAllowFormat
            org.apache.hadoop.hdfs.server.namenode.TestEditLog
            org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
            org.apache.hadoop.hdfs.server.namenode.TestINodeAttributeProvider
            org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777066/HDFS-9038-007.patch
          JIRA Issue HDFS-9038
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 2adaf36d0177 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b7b2923
          findbugs v3.0.0
          javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13835/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt
          javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13835/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13835/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13835/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13835/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13835/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13835/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13835/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
          Max memory used 75MB
          Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13835/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. +1 mvninstall 16m 3s trunk passed +1 compile 4m 56s trunk passed with JDK v1.8.0_66 +1 compile 3m 30s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 45s trunk passed +1 mvnsite 3m 9s trunk passed +1 mvneclipse 0m 51s trunk passed +1 findbugs 7m 36s trunk passed +1 javadoc 3m 46s trunk passed with JDK v1.8.0_66 +1 javadoc 4m 42s trunk passed with JDK v1.7.0_91 +1 mvninstall 3m 0s the patch passed +1 compile 5m 0s the patch passed with JDK v1.8.0_66 +1 cc 5m 0s the patch passed -1 javac 20m 19s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 49). +1 javac 5m 0s the patch passed +1 compile 3m 35s the patch passed with JDK v1.7.0_91 +1 cc 3m 35s the patch passed -1 javac 23m 54s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 51, now 51). +1 javac 3m 35s the patch passed -1 checkstyle 0m 44s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 293, now 295). +1 mvnsite 3m 12s the patch passed +1 mvneclipse 0m 50s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 8m 45s the patch passed +1 javadoc 3m 54s the patch passed with JDK v1.8.0_66 +1 javadoc 4m 55s the patch passed with JDK v1.7.0_91 +1 unit 2m 24s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 105m 50s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 2m 8s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 1m 14s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 36s Patch generated 61 ASF License warnings. 198m 16s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestParallelShortCircuitRead   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.fs.contract.hdfs.TestHDFSContractMkdir   hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens   hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM   hadoop.cli.TestDeleteCLI   hadoop.hdfs.tools.TestDFSZKFailoverController   hadoop.hdfs.TestFileLengthOnClusterRestart   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.cli.TestHDFSCLI   hadoop.hdfs.server.namenode.TestCheckpoint   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.tools.TestGetGroups   hadoop.hdfs.TestRemoteBlockReader2   hadoop.hdfs.server.namenode.TestStartup   hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics   hadoop.hdfs.TestDFSStorageStateRecovery   hadoop.hdfs.server.namenode.TestFSImageWithXAttr   hadoop.hdfs.TestRemoteBlockReader   hadoop.hdfs.TestMultiThreadedHflush   hadoop.fs.contract.hdfs.TestHDFSContractRename   hadoop.hdfs.TestBlockReaderLocal   hadoop.cli.TestCacheAdminCLI   hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits   hadoop.hdfs.server.namenode.TestNameNodeRecovery   hadoop.hdfs.server.namenode.ha.TestFailureOfSharedDir   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.fs.loadGenerator.TestLoadGenerator   hadoop.hdfs.server.namenode.TestFSImageWithAcl   hadoop.hdfs.server.namenode.TestLargeDirectoryDelete   hadoop.fs.TestFcHdfsSetUMask   hadoop.hdfs.TestPread   hadoop.hdfs.server.namenode.ha.TestQuotasWithHA   hadoop.hdfs.crypto.TestHdfsCryptoStreams   hadoop.fs.viewfs.TestViewFsFileStatusHdfs   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl   hadoop.hdfs.TestDFSAddressConfig   hadoop.tracing.TestTracingShortCircuitLocalRead   hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.server.namenode.TestFSDirectory   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.tracing.TestTracing   hadoop.hdfs.TestParallelRead   hadoop.hdfs.TestDFSStripedInputStream   hadoop.hdfs.tools.TestDelegationTokenFetcher   hadoop.hdfs.TestRestartDFS   hadoop.fs.TestWebHdfsFileContextMainOperations   hadoop.hdfs.TestParallelShortCircuitReadNoChecksum   hadoop.hdfs.TestParallelShortCircuitLegacyRead   hadoop.net.TestNetworkTopology   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.fs.TestFcHdfsCreateMkdir   hadoop.hdfs.server.namenode.TestDeadDatanode   hadoop.hdfs.server.namenode.ha.TestNNHealthCheck   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled   hadoop.hdfs.TestFetchImage   hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.TestDFSUpgrade   hadoop.fs.TestFcHdfsPermission   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks   hadoop.hdfs.server.namenode.TestProtectedDirectories   hadoop.hdfs.TestMissingBlocksAlert   hadoop.hdfs.server.namenode.ha.TestHAMetrics   hadoop.hdfs.TestQuota   hadoop.hdfs.server.namenode.TestQuotaByStorageType   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.server.namenode.TestSaveNamespace   hadoop.hdfs.tools.TestDebugAdmin   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover   hadoop.hdfs.server.namenode.TestNameNodeRpcServer   hadoop.hdfs.server.namenode.TestSecureNameNode   hadoop.hdfs.server.namenode.TestSnapshotPathINodes   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer   hadoop.hdfs.TestModTime   hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA   hadoop.hdfs.TestGetBlocks   hadoop.hdfs.TestDFSStartupVersions   hadoop.hdfs.TestRenameWhileOpen   hadoop.fs.viewfs.TestViewFsWithXAttrs   hadoop.cli.TestErasureCodingCLI   hadoop.tools.TestJMXGet   hadoop.hdfs.TestFileAppend4   hadoop.TestGenericRefresh   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.server.namenode.TestAddBlock   hadoop.hdfs.protocol.TestLocatedBlock   hadoop.hdfs.TestDataTransferKeepalive   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.TestFileAppend2   hadoop.hdfs.TestErasureCodeBenchmarkThroughput   hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.TestAuditLogs   hadoop.fs.TestSWebHdfsFileContextMainOperations   hadoop.hdfs.TestInjectionForSimulatedStorage   hadoop.hdfs.server.namenode.TestFsck   hadoop.fs.contract.hdfs.TestHDFSContractDelete   hadoop.fs.contract.hdfs.TestHDFSContractSetTimes   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.server.namenode.TestListCorruptFileBlocks   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr   hadoop.hdfs.TestConnCache   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.server.namenode.TestStripedINodeFile   hadoop.hdfs.TestGetFileChecksum   hadoop.hdfs.TestFileAppend   hadoop.hdfs.TestReservedRawPaths   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.TestRefreshCallQueue   hadoop.hdfs.TestDisableConnCache   hadoop.cli.TestXAttrCLI   hadoop.hdfs.server.namenode.TestMetadataVersionOutput   hadoop.hdfs.TestParallelShortCircuitReadUnCached   hadoop.fs.TestUrlStreamHandler   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020   hadoop.fs.TestEnhancedByteBufferAccess   hadoop.hdfs.TestHDFSServerPorts   hadoop.hdfs.server.namenode.TestAuditLogger   hadoop.hdfs.TestRollingUpgrade   hadoop.fs.viewfs.TestViewFileSystemWithXAttrs   hadoop.fs.TestHDFSFileContextMainOperations   hadoop.hdfs.TestSafeMode   hadoop.hdfs.protocolPB.TestPBHelper   hadoop.hdfs.server.namenode.TestBackupNode   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement   hadoop.hdfs.server.namenode.TestAclConfigFlag   hadoop.hdfs.TestFileStatusWithECPolicy   hadoop.hdfs.TestWriteRead   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.TestExternalBlockReader   hadoop.hdfs.server.namenode.TestMetaSave   hadoop.fs.contract.hdfs.TestHDFSContractAppend   hadoop.hdfs.server.namenode.ha.TestStateTransitionFailure   hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.TestHdfsAdmin   hadoop.fs.contract.hdfs.TestHDFSContractOpen   hadoop.hdfs.server.namenode.TestAddStripedBlocks   hadoop.hdfs.server.namenode.TestGenericJournalConf   hadoop.hdfs.TestFileCreation   hadoop.hdfs.server.mover.TestStorageMover   hadoop.hdfs.TestClientReportBadBlock   hadoop.tracing.TestTraceAdmin   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.server.namenode.TestBlockUnderConstruction   hadoop.hdfs.TestFileCorruption   hadoop.hdfs.server.namenode.TestHDFSConcat   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.TestClose   hadoop.hdfs.TestAbandonBlock   hadoop.hdfs.server.namenode.ha.TestHAFsck   hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant   hadoop.hdfs.TestDFSInputStream   hadoop.hdfs.TestLeaseRecovery   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.TestWriteBlockGetsBlockLengthHint   hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages   hadoop.fs.viewfs.TestViewFsHdfs   hadoop.hdfs.TestDFSClientSocketSize   hadoop.fs.contract.hdfs.TestHDFSContractConcat   hadoop.hdfs.server.namenode.TestFileLimit   hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot   hadoop.hdfs.TestHDFSTrash   hadoop.hdfs.TestSmallBlock   hadoop.cli.TestCryptoAdminCLI   hadoop.hdfs.TestFileCreationDelete   hadoop.hdfs.server.namenode.ha.TestLossyRetryInvocationHandler   hadoop.hdfs.tools.TestDFSHAAdminMiniCluster   hadoop.hdfs.TestReplication   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.cli.TestAclCLI   hadoop.fs.viewfs.TestViewFileSystemWithAcls   hadoop.hdfs.server.namenode.TestEditLogAutoroll   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.server.namenode.TestTransferFsImage   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.fs.TestSymlinkHdfsFileSystem   hadoop.hdfs.TestFileAppend3   hadoop.fs.TestUnbuffer   hadoop.hdfs.TestPipelines   hadoop.hdfs.TestErasureCodingPolicies   hadoop.hdfs.TestAclsEndToEnd   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.ha.TestHASafeMode   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.TestBlockReaderFactory   hadoop.hdfs.server.namenode.ha.TestStandbyIsHot   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestFSOutputSummer   hadoop.hdfs.TestBlockReaderLocalLegacy   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   hadoop.fs.viewfs.TestViewFsWithAcls   hadoop.hdfs.server.namenode.ha.TestHAStateTransitions   hadoop.hdfs.TestEncryptionZones   hadoop.fs.TestResolveHdfsSymlink   hadoop.hdfs.server.namenode.TestCommitBlockWithInvalidGenStamp   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider   hadoop.hdfs.TestFileConcurrentReader   hadoop.fs.viewfs.TestViewFsDefaultValue   hadoop.hdfs.TestAppendDifferentChecksum   hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus   hadoop.hdfs.server.namenode.TestCreateEditsLog   hadoop.hdfs.server.namenode.TestFSNamesystemMBean   hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA   hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary   hadoop.hdfs.server.namenode.TestNameNodeXAttr   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.fs.viewfs.TestViewFileSystemHdfs   hadoop.hdfs.TestParallelUnixDomainRead   hadoop.hdfs.tools.TestStoragePolicyCommands   hadoop.hdfs.TestDFSRemove   hadoop.fs.viewfs.TestViewFsAtHdfsRoot   hadoop.hdfs.qjournal.TestNNWithQJM   hadoop.hdfs.TestFileCreationClient   hadoop.hdfs.TestRollingUpgradeDowngrade   hadoop.fs.contract.hdfs.TestHDFSContractCreate   hadoop.fs.permission.TestStickyBit   hadoop.hdfs.TestEncryptionZonesWithKMS   hadoop.fs.TestSymlinkHdfsDisable   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.namenode.TestDeleteRace   hadoop.hdfs.TestLease   hadoop.hdfs.TestLargeBlock   hadoop.fs.TestGlobPaths   hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes   hadoop.hdfs.server.namenode.TestEditLogJournalFailures   hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.TestSecondaryWebUi   hadoop.fs.contract.hdfs.TestHDFSContractSeek   hadoop.fs.shell.TestHdfsTextCommand   hadoop.hdfs.TestBlockMissingException JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestAllowFormat   org.apache.hadoop.hdfs.server.namenode.TestEditLog   org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   org.apache.hadoop.hdfs.server.namenode.TestINodeAttributeProvider   org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777066/HDFS-9038-007.patch JIRA Issue HDFS-9038 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 2adaf36d0177 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b7b2923 findbugs v3.0.0 javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13835/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13835/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13835/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13835/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13835/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13835/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13835/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13835/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 75MB Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13835/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          +1 mvninstall 7m 49s trunk passed
          +1 compile 1m 30s trunk passed with JDK v1.8.0_66
          +1 compile 1m 35s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 1m 27s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 50s trunk passed
          +1 javadoc 1m 31s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 15s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 23s the patch passed
          +1 compile 1m 31s the patch passed with JDK v1.8.0_66
          +1 cc 1m 31s the patch passed
          -1 javac 8m 39s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 49).
          +1 javac 1m 31s the patch passed
          +1 compile 1m 36s the patch passed with JDK v1.7.0_91
          +1 cc 1m 36s the patch passed
          -1 javac 10m 16s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 51, now 51).
          +1 javac 1m 36s the patch passed
          -1 checkstyle 0m 22s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 293, now 295).
          +1 mvnsite 1m 26s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 7s the patch passed
          +1 javadoc 1m 37s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 22s the patch passed with JDK v1.7.0_91
          +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          +1 unit 68m 27s hadoop-hdfs in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          -1 unit 68m 35s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 22s Patch generated 58 ASF License warnings.
          177m 44s



          Reason Tests
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDataTransferProtocol
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777066/HDFS-9038-007.patch
          JIRA Issue HDFS-9038
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 6f155179183b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 576b569
          findbugs v3.0.0
          javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13836/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt
          javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13836/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13836/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13836/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13836/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13836/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13836/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
          Max memory used 75MB
          Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13836/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. +1 mvninstall 7m 49s trunk passed +1 compile 1m 30s trunk passed with JDK v1.8.0_66 +1 compile 1m 35s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 50s trunk passed +1 javadoc 1m 31s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 15s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 23s the patch passed +1 compile 1m 31s the patch passed with JDK v1.8.0_66 +1 cc 1m 31s the patch passed -1 javac 8m 39s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 49). +1 javac 1m 31s the patch passed +1 compile 1m 36s the patch passed with JDK v1.7.0_91 +1 cc 1m 36s the patch passed -1 javac 10m 16s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 51, now 51). +1 javac 1m 36s the patch passed -1 checkstyle 0m 22s Patch generated 4 new checkstyle issues in hadoop-hdfs-project (total was 293, now 295). +1 mvnsite 1m 26s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 7s the patch passed +1 javadoc 1m 37s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 22s the patch passed with JDK v1.7.0_91 +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. +1 unit 68m 27s hadoop-hdfs in the patch passed with JDK v1.8.0_66. +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 68m 35s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 22s Patch generated 58 ASF License warnings. 177m 44s Reason Tests JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDataTransferProtocol   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777066/HDFS-9038-007.patch JIRA Issue HDFS-9038 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 6f155179183b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 576b569 findbugs v3.0.0 javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13836/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13836/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13836/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13836/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13836/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13836/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13836/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 75MB Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13836/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          dfs.datanode.du.reserved means that the administrator intentionally wants to hold back a portion of the volume, essentially making it invisible to HDFS. This is the same reason that dfs.datanode.du.reserved gets subtracted from the capacity calculation. This is why I think 3 GB is incorrect in the example above. Non-DFS usage is really "unexpected non-DFS usage". By setting dfs.datanode.du.reserved to 1 GB, you have stated that up to 1 GB of usage by something other than HDFS is expected, so it's incorrect to count it as part of "unexpected non-DFS usage".

          So you mean, when 1GB is configured as reserved, 'Non-DFS' usage metric should show 2GB, even though actual usage by files other than HDFS is 3GB.?
          I was thinking 'NonDfsUsage' should show actual non-dfs usage. Not only unexpected part of it. So if actual non-dfs usage is within this reserved limit, metric should show 0. right?

          In that case, wouldn't it be better to rename the 'NonDfsUsage' metric to 'UnexpectedNonDfsUsage' to make it more clear? Or mention somewhere to clear confusion in people like me.

          Show
          vinayrpet Vinayakumar B added a comment - dfs.datanode.du.reserved means that the administrator intentionally wants to hold back a portion of the volume, essentially making it invisible to HDFS. This is the same reason that dfs.datanode.du.reserved gets subtracted from the capacity calculation. This is why I think 3 GB is incorrect in the example above. Non-DFS usage is really "unexpected non-DFS usage". By setting dfs.datanode.du.reserved to 1 GB, you have stated that up to 1 GB of usage by something other than HDFS is expected, so it's incorrect to count it as part of "unexpected non-DFS usage". So you mean, when 1GB is configured as reserved, 'Non-DFS' usage metric should show 2GB, even though actual usage by files other than HDFS is 3GB.? I was thinking 'NonDfsUsage' should show actual non-dfs usage. Not only unexpected part of it. So if actual non-dfs usage is within this reserved limit, metric should show 0. right? In that case, wouldn't it be better to rename the 'NonDfsUsage' metric to 'UnexpectedNonDfsUsage' to make it more clear? Or mention somewhere to clear confusion in people like me.
          Hide
          cnauroth Chris Nauroth added a comment -

          So if actual non-dfs usage is within this reserved limit, metric should show 0. right?

          Yes, that's my understanding, and that's the behavior pre-HDFS-5215.

          In that case, wouldn't it be better to rename the 'NonDfsUsage' metric to 'UnexpectedNonDfsUsage' to make it more clear? Or mention somewhere to clear confusion in people like me.

          I don' t think it could be renamed easily due to backwards-compatibility, but I do think we could update Metrics.md.

          Show
          cnauroth Chris Nauroth added a comment - So if actual non-dfs usage is within this reserved limit, metric should show 0. right? Yes, that's my understanding, and that's the behavior pre- HDFS-5215 . In that case, wouldn't it be better to rename the 'NonDfsUsage' metric to 'UnexpectedNonDfsUsage' to make it more clear? Or mention somewhere to clear confusion in people like me. I don' t think it could be renamed easily due to backwards-compatibility, but I do think we could update Metrics.md.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          Brahma, thanks for your patience as we work through the math. It may be useful to describe your proposed derivation as a Jira comment before you post another patch.

          Chris Nauroth I think you are right in conclusion but there is a misstep in the equations in this comment. reserved should cancel out and it should not factor in the final computation.

          The problem with the v005 patch we missed earlier is that getCapacity() subtracts reserved space. We should use the raw capacity.

          One way to fix it is:

          public long getNonDfsUsed() throws IOException {
            long totalFreeSpace = currentDir.getFreeSpace();
            long nonDfsUsed = getCapacity() + reserved - getDfsUsed() - totalFreeSpace;
            return (nonDfsUsed >= 0) ? nonDfsUsed : 0;
          }
          

          Then the derivation becomes:

          1: non-DFS used = getCapacity() + reserved - getDfsUsed() - totalFreeSpace
          2:              = usage.getCapacity() - reserved + reserved - getDfsUsed() - totalFreeSpace
          3:              = usage.getCapacity() - getDfsUsed() - totalFreeSpace
          4:              = File#getTotalSpace - getDfsUsed() - File#getFreeSpace
          

          Hope that makes sense.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited Brahma , thanks for your patience as we work through the math. It may be useful to describe your proposed derivation as a Jira comment before you post another patch. Chris Nauroth I think you are right in conclusion but there is a misstep in the equations in this comment . reserved should cancel out and it should not factor in the final computation. The problem with the v005 patch we missed earlier is that getCapacity() subtracts reserved space. We should use the raw capacity. One way to fix it is: public long getNonDfsUsed() throws IOException { long totalFreeSpace = currentDir.getFreeSpace(); long nonDfsUsed = getCapacity() + reserved - getDfsUsed() - totalFreeSpace; return (nonDfsUsed >= 0) ? nonDfsUsed : 0; } Then the derivation becomes: 1: non-DFS used = getCapacity() + reserved - getDfsUsed() - totalFreeSpace 2: = usage.getCapacity() - reserved + reserved - getDfsUsed() - totalFreeSpace 3: = usage.getCapacity() - getDfsUsed() - totalFreeSpace 4: = File#getTotalSpace - getDfsUsed() - File#getFreeSpace Hope that makes sense.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Arpit Agarwal, thanks for jumping in for calculation.

          I think considering We use File#getUsableSpace instead of File#getFreeSpace, current code after HDFS-5215 also comes to same equation as mentioned by Arpit Agarwal, brahma's patch doesn't change anything in this, except subtracting 'reservedForReplicas' also.

          But, I think what Chris Nauroth expects is,
          Since reserved is hidden from the HDFS in getCapacity() itself, we can think that's already used for nonDfs and subtract it from actual ondisk nonDfsUsage, showing only HDFS visible nonDfsUsage. nonDfsUsage metric will be positive, only if the ondisk nonDfsUsage crosses beyond reserved, and it shows only excess usage beyond reserved.
          i.e. nonDfsUsage will be 2GB instead of 3GB in my earlier example, where 1GB was reserved.

          So final equation would be

          nonDfsUsage=File#getTotalSpace - dfsUsed - File#getUsableSpace - reserved

          Code would look like this.

            public long getNonDfsUsed() throws IOException {
              long nonDfsUsed = getCapacity() - getDfsUsed() - getAvailable() - reserved - getReservedForReplicas();
              return (nonDfsUsed > 0) ? nonDfsUsed : 0;
            }
          

          Am I right Chris Nauroth?

          Show
          vinayrpet Vinayakumar B added a comment - Arpit Agarwal , thanks for jumping in for calculation. I think considering We use File#getUsableSpace instead of File#getFreeSpace , current code after HDFS-5215 also comes to same equation as mentioned by Arpit Agarwal , brahma's patch doesn't change anything in this, except subtracting 'reservedForReplicas' also. But, I think what Chris Nauroth expects is, Since reserved is hidden from the HDFS in getCapacity() itself, we can think that's already used for nonDfs and subtract it from actual ondisk nonDfsUsage , showing only HDFS visible nonDfsUsage . nonDfsUsage metric will be positive, only if the ondisk nonDfsUsage crosses beyond reserved , and it shows only excess usage beyond reserved . i.e. nonDfsUsage will be 2GB instead of 3GB in my earlier example, where 1GB was reserved. So final equation would be nonDfsUsage=File#getTotalSpace - dfsUsed - File#getUsableSpace - reserved Code would look like this. public long getNonDfsUsed() throws IOException { long nonDfsUsed = getCapacity() - getDfsUsed() - getAvailable() - reserved - getReservedForReplicas(); return (nonDfsUsed > 0) ? nonDfsUsed : 0; } Am I right Chris Nauroth ?
          Hide
          cnauroth Chris Nauroth added a comment -

          Thanks everyone for sticking with this. This has turned out to be much trickier than I anticipated when I filed the issue. I'd like to summarize current status.

          Arpit and I are in agreement about my analysis of how the calculation changed after HDFS-5215. However, we are not yet in agreement about which calculation is truly correct. I believe the pre-HDFS-5215 calculation (subtracting dfs.datanode.du.reserved) is correct, because it allowed me to monitor for unexpected non-zero non-DFS usage and react. Since this was an established operations workflow (at least for me), I argue that we have a responsibility to restore that behavior. Arpit believes that it's correct to cancel out dfs.datanode.du.reserved, because then non-DFS used would report space used for non-HDFS purposes more accurately. Essentially, it's a question of whether this metric means "Raw Non-DFS Used" or "Unplanned Non-DFS Used".

          We also discovered an interesting side issue about File#getUsableSpace vs. File#getFreeSpace. Pre-HDFS-5215, it could be considered a bug that we did not account for system reserved space. Interestingly, it seems in our testing that ext holds back 5% by default, but xfs does not.

          I pushed pretty hard for restoring the pre-HDFS-5215 behavior in my earlier comments, but I'm just one voice. I suggest that we leave this issue open for a while for others to comment. I could be swayed if others think I'm approaching this incorrectly. Meanwhile, Brahma Reddy Battula, would you please hold off on posting more patches? Let's wait for the discussion to settle a little more first. Thanks for your patience.

          Show
          cnauroth Chris Nauroth added a comment - Thanks everyone for sticking with this. This has turned out to be much trickier than I anticipated when I filed the issue. I'd like to summarize current status. Arpit and I are in agreement about my analysis of how the calculation changed after HDFS-5215 . However, we are not yet in agreement about which calculation is truly correct. I believe the pre- HDFS-5215 calculation (subtracting dfs.datanode.du.reserved ) is correct, because it allowed me to monitor for unexpected non-zero non-DFS usage and react. Since this was an established operations workflow (at least for me), I argue that we have a responsibility to restore that behavior. Arpit believes that it's correct to cancel out dfs.datanode.du.reserved , because then non-DFS used would report space used for non-HDFS purposes more accurately. Essentially, it's a question of whether this metric means "Raw Non-DFS Used" or "Unplanned Non-DFS Used". We also discovered an interesting side issue about File#getUsableSpace vs. File#getFreeSpace . Pre- HDFS-5215 , it could be considered a bug that we did not account for system reserved space. Interestingly, it seems in our testing that ext holds back 5% by default, but xfs does not. I pushed pretty hard for restoring the pre- HDFS-5215 behavior in my earlier comments, but I'm just one voice. I suggest that we leave this issue open for a while for others to comment. I could be swayed if others think I'm approaching this incorrectly. Meanwhile, Brahma Reddy Battula , would you please hold off on posting more patches? Let's wait for the discussion to settle a little more first. Thanks for your patience.
          Hide
          aw Allen Wittenauer added a comment -

          I'd really like for someone to attach a simple test case so that we can run and see the differences on different file systems. In particular, every time we change this code, we only ever test on one or two Linux file systems and end up causing really oddball behaviors on others, especially those that don't use the traditional disk->partition->filesystem layout (e.g., ZFS).

          Show
          aw Allen Wittenauer added a comment - I'd really like for someone to attach a simple test case so that we can run and see the differences on different file systems. In particular, every time we change this code, we only ever test on one or two Linux file systems and end up causing really oddball behaviors on others, especially those that don't use the traditional disk->partition->filesystem layout (e.g., ZFS).
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          tl; dr - We need agreement on the definition of non-DFS used.

          The pre HDFS-5215 calculation had two bugs.

          1. It incorrectly subtracted reserved space from the non-DFS used. (net negative). Chris suggests this is not really an issue as non-DFS used should be shown as zero unless it exceeds the DFS reserved value.
          2. It used File#getUsableSpace to calculate the volume free space instead of File#getFreeSpace. (net positive)

          The net effect was that non-DFS used was displayed as zero unless the actual non-DFS used exceeded DFS reserved - system reserved.

          HDFS-5215 fixed the first issue and the value that is now erroneously counted towards non-DFS used is in fact the system reserved 5%.

          Also attached a trivial utility that dumps the free/available space.

          From a mostly empty 40GB Ext4 partition:

          $ java GetFree /mnt/sdb/hadoop/
          Free space       : 42,090,229,760
          Available space  : 39,925,968,896
          

          Same partition reformatted as XFS:

          Free space       : 42,894,983,168
          Available space  : 42,894,983,168
          

          So Ext derivatives hold back 5% free space while XFS does not.

          Edit: Removed statement about Jira description being inaccurate since it too depends on how we define non-DFS used.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited tl; dr - We need agreement on the definition of non-DFS used. The pre HDFS-5215 calculation had two bugs. It incorrectly subtracted reserved space from the non-DFS used. (net negative). Chris suggests this is not really an issue as non-DFS used should be shown as zero unless it exceeds the DFS reserved value. It used File#getUsableSpace to calculate the volume free space instead of File#getFreeSpace. (net positive) The net effect was that non-DFS used was displayed as zero unless the actual non-DFS used exceeded DFS reserved - system reserved . HDFS-5215 fixed the first issue and the value that is now erroneously counted towards non-DFS used is in fact the system reserved 5%. Also attached a trivial utility that dumps the free/available space. From a mostly empty 40GB Ext4 partition: $ java GetFree /mnt/sdb/hadoop/ Free space : 42,090,229,760 Available space : 39,925,968,896 Same partition reformatted as XFS: Free space : 42,894,983,168 Available space : 42,894,983,168 So Ext derivatives hold back 5% free space while XFS does not. Edit: Removed statement about Jira description being inaccurate since it too depends on how we define non-DFS used.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Hi Uma Maheswara Rao G/Yi Liu/Tsz Wo Nicholas Sze/Haohui Mai,
          Do You have any opinions to here?

          Show
          vinayrpet Vinayakumar B added a comment - Hi Uma Maheswara Rao G / Yi Liu / Tsz Wo Nicholas Sze / Haohui Mai , Do You have any opinions to here?
          Hide
          vinayrpet Vinayakumar B added a comment -

          Ping.
          Any update on this guys?

          Show
          vinayrpet Vinayakumar B added a comment - Ping. Any update on this guys?
          Hide
          vinayrpet Vinayakumar B added a comment -
          Show
          vinayrpet Vinayakumar B added a comment - Ping Chris Nauroth .
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          I suggest that we leave this issue open for a while for others to comment. I could be swayed if others think I'm approaching this incorrectly

          Chris Nauroth/Arpit Agarwal/Vinayakumar B can we discuss in mailing list Or do you want specific people to comment here.. Please let me know..

          Show
          brahmareddy Brahma Reddy Battula added a comment - I suggest that we leave this issue open for a while for others to comment. I could be swayed if others think I'm approaching this incorrectly Chris Nauroth / Arpit Agarwal / Vinayakumar B can we discuss in mailing list Or do you want specific people to comment here.. Please let me know..
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          Hi Brahma Reddy Battula, I am sorry this patch is stuck after all the effort you put into it.

          There is a disagreement on the definition of non-DFS used space. Quoting Chris Nauroth:

          Essentially, it's a question of whether this metric means "Raw Non-DFS Used" or "Unplanned Non-DFS Used".

          I don't see any harm in starting a discussion on hdfs-dev. I will think about this some more.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited Hi Brahma Reddy Battula , I am sorry this patch is stuck after all the effort you put into it. There is a disagreement on the definition of non-DFS used space. Quoting Chris Nauroth : Essentially, it's a question of whether this metric means "Raw Non-DFS Used" or "Unplanned Non-DFS Used". I don't see any harm in starting a discussion on hdfs-dev. I will think about this some more.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          it's a question of whether this metric means "Raw Non-DFS Used" or "Unplanned Non-DFS Used".

          I prefer (I'm +0 for) the following calculation, which is straightforward for me.

          Non-DFS used = f.getTotalSpace() - f.getFreeSpace() - getDfsUsed()
          

          When we reach an agreement, it should be well-documented.

          Show
          ajisakaa Akira Ajisaka added a comment - it's a question of whether this metric means "Raw Non-DFS Used" or "Unplanned Non-DFS Used". I prefer (I'm +0 for) the following calculation, which is straightforward for me. Non-DFS used = f.getTotalSpace() - f.getFreeSpace() - getDfsUsed() When we reach an agreement, it should be well-documented.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          I spent a few hours looking at this today. I now think that the current logic in FsVolumeImpl#getAvailable is too pessimistic and the impact of bug may be more severe than reported.

          public long getAvailable() throws IOException {
            long remaining = getCapacity() - getDfsUsed() - reservedForReplicas.get();
            long available = usage.getAvailable() - reserved
                - reservedForReplicas.get();
            if (remaining > available) {
              remaining = available;
            }
            return (remaining > 0) ? remaining : 0;
          }
          

          Specifically {{available = usage.getAvailable() - reserved ... }} can account for the non-DFS usage twice. e.g. say 10GB is reserved and YARN is using all of that 10GB. usage.getAvailable() has already accounted for that 10GB and further subtracting it makes us underestimate available.

          Instead the calculation should only subtract the unused portion of non-DFS used. What do you guys think?

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited I spent a few hours looking at this today. I now think that the current logic in FsVolumeImpl#getAvailable is too pessimistic and the impact of bug may be more severe than reported. public long getAvailable() throws IOException { long remaining = getCapacity() - getDfsUsed() - reservedForReplicas.get(); long available = usage.getAvailable() - reserved - reservedForReplicas.get(); if (remaining > available) { remaining = available; } return (remaining > 0) ? remaining : 0; } Specifically {{available = usage.getAvailable() - reserved ... }} can account for the non-DFS usage twice. e.g. say 10GB is reserved and YARN is using all of that 10GB. usage.getAvailable() has already accounted for that 10GB and further subtracting it makes us underestimate available. Instead the calculation should only subtract the unused portion of non-DFS used. What do you guys think?
          Hide
          vinayrpet Vinayakumar B added a comment -

          Instead the calculation should only subtract the unused portion of non-DFS used. What do you guys think?

          I think below code change suggested by brahma in the mailing list does exactly what you suggesting. Right?

             public long getAvailable() throws IOException {
               long remaining = getCapacity() - getDfsUsed() - reservedForReplicas.get();
          -    long available = usage.getAvailable() - reserved
          +    long available = usage.getAvailable() - getRemainingReserved()
                   - reservedForReplicas.get();
               if (remaining > available) {
                 remaining = available;
          @@ -391,6 +391,31 @@ public long getAvailable() throws IOException {
               return (remaining > 0) ? remaining : 0;
             }
          
          +  private long getActualNonDfsUsed() throws IOException {
          +    return usage.getUsed() - getDfsUsed();
          +  }
          +
          +  private long getRemainingReserved() throws IOException {
          +    long actualNonDfsUsed = getActualNonDfsUsed();
          +    if (actualNonDfsUsed < reserved) {
          +      return reserved - actualNonDfsUsed;
          +    }
          +    return 0L;
          +  }
          +
          +  /**
          +   * Unplanned Non-DFS usage, i.e. Extra usage beyond reserved.
          +   * @return
          +   * @throws IOException
          +   */
          +  public long getNonDfsUsed() throws IOException {
          +    long actualNonDfsUsed = getActualNonDfsUsed();
          +    if (actualNonDfsUsed < reserved) {
          +      return 0L;
          +    }
          +    return actualNonDfsUsed - reserved;
          +  }
          +
          
          Show
          vinayrpet Vinayakumar B added a comment - Instead the calculation should only subtract the unused portion of non-DFS used. What do you guys think? I think below code change suggested by brahma in the mailing list does exactly what you suggesting. Right? public long getAvailable() throws IOException { long remaining = getCapacity() - getDfsUsed() - reservedForReplicas.get(); - long available = usage.getAvailable() - reserved + long available = usage.getAvailable() - getRemainingReserved() - reservedForReplicas.get(); if (remaining > available) { remaining = available; @@ -391,6 +391,31 @@ public long getAvailable() throws IOException { return (remaining > 0) ? remaining : 0; } + private long getActualNonDfsUsed() throws IOException { + return usage.getUsed() - getDfsUsed(); + } + + private long getRemainingReserved() throws IOException { + long actualNonDfsUsed = getActualNonDfsUsed(); + if (actualNonDfsUsed < reserved) { + return reserved - actualNonDfsUsed; + } + return 0L; + } + + /** + * Unplanned Non-DFS usage, i.e. Extra usage beyond reserved. + * @ return + * @ throws IOException + */ + public long getNonDfsUsed() throws IOException { + long actualNonDfsUsed = getActualNonDfsUsed(); + if (actualNonDfsUsed < reserved) { + return 0L; + } + return actualNonDfsUsed - reserved; + } +
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks Vinay. I see Tsz Wo Nicholas Sze already brought this up on the mailing list thread. Brahma's proposed update looks good to me.

          Will this delta be applied on top of the v7 patch?

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks Vinay. I see Tsz Wo Nicholas Sze already brought this up on the mailing list thread. Brahma's proposed update looks good to me. Will this delta be applied on top of the v7 patch?
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Cool.. I will update v7 patch. thanks vinay and arpit..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Cool.. I will update v7 patch. thanks vinay and arpit..
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Uploaded the patch.. Arpit Agarwal and Vinayakumar B kindly review.. and sorry for long delay.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Uploaded the patch.. Arpit Agarwal and Vinayakumar B kindly review.. and sorry for long delay.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          0 mvndep 0m 30s Maven dependency ordering for branch
          +1 mvninstall 6m 55s trunk passed
          +1 compile 1m 24s trunk passed
          +1 checkstyle 0m 38s trunk passed
          +1 mvnsite 1m 25s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 3m 8s trunk passed
          +1 javadoc 1m 18s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 18s the patch passed
          +1 compile 1m 23s the patch passed
          +1 cc 1m 23s the patch passed
          -1 javac 1m 23s hadoop-hdfs-project generated 1 new + 51 unchanged - 1 fixed = 52 total (was 52)
          -0 checkstyle 0m 33s hadoop-hdfs-project: The patch generated 7 new + 540 unchanged - 2 fixed = 547 total (was 542)
          +1 mvnsite 1m 25s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 36s the patch passed
          +1 javadoc 1m 12s the patch passed
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed.
          -1 unit 58m 20s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          86m 46s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.tracing.TestTracing
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822051/HDFS-9038-008.patch
          JIRA Issue HDFS-9038
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 039d0e6011f7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 70c2781
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/16316/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16316/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16316/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16316/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16316/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. 0 mvndep 0m 30s Maven dependency ordering for branch +1 mvninstall 6m 55s trunk passed +1 compile 1m 24s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 1m 25s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 3m 8s trunk passed +1 javadoc 1m 18s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 18s the patch passed +1 compile 1m 23s the patch passed +1 cc 1m 23s the patch passed -1 javac 1m 23s hadoop-hdfs-project generated 1 new + 51 unchanged - 1 fixed = 52 total (was 52) -0 checkstyle 0m 33s hadoop-hdfs-project: The patch generated 7 new + 540 unchanged - 2 fixed = 547 total (was 542) +1 mvnsite 1m 25s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 36s the patch passed +1 javadoc 1m 12s the patch passed +1 unit 0m 54s hadoop-hdfs-client in the patch passed. -1 unit 58m 20s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 86m 46s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.tracing.TestTracing   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822051/HDFS-9038-008.patch JIRA Issue HDFS-9038 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 039d0e6011f7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 70c2781 Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/16316/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16316/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16316/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16316/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16316/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Uploaded the patch to fix checkstyle issues..Testcases are passing locally..Seems to be there is a difference in jenkins and local run...

          Show
          brahmareddy Brahma Reddy Battula added a comment - Uploaded the patch to fix checkstyle issues..Testcases are passing locally..Seems to be there is a difference in jenkins and local run...
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 59s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
          0 mvndep 0m 29s Maven dependency ordering for branch
          +1 mvninstall 7m 23s trunk passed
          +1 compile 1m 55s trunk passed
          +1 checkstyle 0m 42s trunk passed
          +1 mvnsite 1m 53s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 31s trunk passed
          +1 javadoc 1m 20s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 44s the patch passed
          +1 compile 1m 48s the patch passed
          +1 cc 1m 48s the patch passed
          -1 javac 1m 48s hadoop-hdfs-project generated 1 new + 51 unchanged - 1 fixed = 52 total (was 52)
          -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 2 new + 594 unchanged - 5 fixed = 596 total (was 599)
          +1 mvnsite 1m 49s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 45s the patch passed
          +1 javadoc 1m 15s the patch passed
          +1 unit 0m 58s hadoop-hdfs-client in the patch passed.
          -1 unit 76m 40s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          109m 41s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.tracing.TestTracing
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.TestFileCreationDelete
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.security.TestRefreshUserMappings



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822418/HDFS-9038-009.patch
          JIRA Issue HDFS-9038
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 16bca1a0834f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 10ed06a
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/16333/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16333/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16333/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16333/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16333/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 59s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 0m 29s Maven dependency ordering for branch +1 mvninstall 7m 23s trunk passed +1 compile 1m 55s trunk passed +1 checkstyle 0m 42s trunk passed +1 mvnsite 1m 53s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 31s trunk passed +1 javadoc 1m 20s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 44s the patch passed +1 compile 1m 48s the patch passed +1 cc 1m 48s the patch passed -1 javac 1m 48s hadoop-hdfs-project generated 1 new + 51 unchanged - 1 fixed = 52 total (was 52) -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 2 new + 594 unchanged - 5 fixed = 596 total (was 599) +1 mvnsite 1m 49s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 45s the patch passed +1 javadoc 1m 15s the patch passed +1 unit 0m 58s hadoop-hdfs-client in the patch passed. -1 unit 76m 40s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 109m 41s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.tracing.TestTracing   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.TestFileCreationDelete   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.security.TestRefreshUserMappings Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822418/HDFS-9038-009.patch JIRA Issue HDFS-9038 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 16bca1a0834f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 10ed06a Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/16333/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16333/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16333/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16333/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16333/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Brahma Reddy Battula, thanks for the updated patch. I will try to review this next week.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Brahma Reddy Battula , thanks for the updated patch. I will try to review this next week.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.7.3 is under release process, changing target-version to 2.7.4.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.7.3 is under release process, changing target-version to 2.7.4.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Uploaded the patch to fix the checkstyle issue and testcases fix. As ext filesystems will reserve 5% space which is not considered in non-dfs,hence TestNamenodeCapacityReport and TestNameNodeMetrics are failing..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Uploaded the patch to fix the checkstyle issue and testcases fix. As ext filesystems will reserve 5% space which is not considered in non-dfs,hence TestNamenodeCapacityReport and TestNameNodeMetrics are failing..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
          0 mvndep 0m 6s Maven dependency ordering for branch
          +1 mvninstall 7m 17s trunk passed
          +1 compile 1m 48s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 2m 1s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 3m 38s trunk passed
          +1 javadoc 1m 22s trunk passed
          0 mvndep 0m 6s Maven dependency ordering for patch
          +1 mvninstall 1m 36s the patch passed
          +1 compile 1m 52s the patch passed
          +1 cc 1m 52s the patch passed
          -1 javac 1m 52s hadoop-hdfs-project generated 1 new + 51 unchanged - 1 fixed = 52 total (was 52)
          -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 1 new + 583 unchanged - 5 fixed = 584 total (was 588)
          +1 mvnsite 1m 46s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 59s the patch passed
          +1 javadoc 1m 12s the patch passed
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed.
          -1 unit 77m 0s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          109m 7s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.mover.TestStorageMover



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825464/HDFS-9038-010.patch
          JIRA Issue HDFS-9038
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 4f982e9d4706 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 525d52b
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/16537/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16537/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16537/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16537/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16537/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 0m 6s Maven dependency ordering for branch +1 mvninstall 7m 17s trunk passed +1 compile 1m 48s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 2m 1s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 3m 38s trunk passed +1 javadoc 1m 22s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 1m 36s the patch passed +1 compile 1m 52s the patch passed +1 cc 1m 52s the patch passed -1 javac 1m 52s hadoop-hdfs-project generated 1 new + 51 unchanged - 1 fixed = 52 total (was 52) -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 1 new + 583 unchanged - 5 fixed = 584 total (was 588) +1 mvnsite 1m 46s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 59s the patch passed +1 javadoc 1m 12s the patch passed +1 unit 0m 57s hadoop-hdfs-client in the patch passed. -1 unit 77m 0s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 109m 7s Reason Tests Failed junit tests hadoop.hdfs.server.mover.TestStorageMover Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825464/HDFS-9038-010.patch JIRA Issue HDFS-9038 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 4f982e9d4706 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 525d52b Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/16537/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16537/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16537/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16537/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16537/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Arpit Agarwal can you please look into latest patch..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Arpit Agarwal can you please look into latest patch..
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for sticking with this difficult fix Brahma Reddy Battula. The patch looks good to me (I am still reviewing unit tests). Nitpick - convert(DatanodeInfoProto di) can use the new DataNodeInfo constructor.

          Vinayakumar B, Chris Nauroth, Tsz Wo Nicholas Sze, do you have any comments on the latest patch.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for sticking with this difficult fix Brahma Reddy Battula . The patch looks good to me (I am still reviewing unit tests). Nitpick - convert(DatanodeInfoProto di) can use the new DataNodeInfo constructor. Vinayakumar B , Chris Nauroth , Tsz Wo Nicholas Sze , do you have any comments on the latest patch.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Arpit Agarwal thanks for review.. will update patch once your review completed..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Arpit Agarwal thanks for review.. will update patch once your review completed..
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          The tests also look fine to me. Thanks Brahma Reddy Battula. We should hold off committing for until next week in case Chris or Vinayakumar want to take a look at the new patch since they reviewed earlier revisions.

          It'd be great to have a targeted unit test for PBHelperClient#convert(DatanodeInfoProto di) that verifies deserialization of protobufs with and without the new non-dfs used field.

          Show
          arpitagarwal Arpit Agarwal added a comment - The tests also look fine to me. Thanks Brahma Reddy Battula . We should hold off committing for until next week in case Chris or Vinayakumar want to take a look at the new patch since they reviewed earlier revisions. It'd be great to have a targeted unit test for PBHelperClient#convert(DatanodeInfoProto di) that verifies deserialization of protobufs with and without the new non-dfs used field.
          Hide
          cnauroth Chris Nauroth added a comment -

          Brahma Reddy Battula, Arpit Agarwal and Vinayakumar B, thank you for your dedication working through this issue. I am +1 to proceed with the patch after addressing Arpit's last round of feedback.

          Show
          cnauroth Chris Nauroth added a comment - Brahma Reddy Battula , Arpit Agarwal and Vinayakumar B , thank you for your dedication working through this issue. I am +1 to proceed with the patch after addressing Arpit's last round of feedback.
          Hide
          brahmareddy Brahma Reddy Battula added a comment - - edited

          Attaching the updated patch with new test.

          Nitpick - convert(DatanodeInfoProto di) can use the new DataNodeInfo constructor

          This constructor was used intentionally as new constructor was expecting all fields to be specified, instead of just DatanodeID

          Please review.

          Show
          brahmareddy Brahma Reddy Battula added a comment - - edited Attaching the updated patch with new test. Nitpick - convert(DatanodeInfoProto di) can use the new DataNodeInfo constructor This constructor was used intentionally as new constructor was expecting all fields to be specified, instead of just DatanodeID Please review.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 10 new or modified test files.
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 7m 7s trunk passed
          +1 compile 1m 35s trunk passed
          +1 checkstyle 0m 45s trunk passed
          +1 mvnsite 1m 34s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 19s trunk passed
          +1 javadoc 1m 18s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 21s the patch passed
          +1 compile 1m 26s the patch passed
          +1 cc 1m 26s the patch passed
          -1 javac 1m 26s hadoop-hdfs-project generated 1 new + 51 unchanged - 1 fixed = 52 total (was 52)
          -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 1 new + 639 unchanged - 5 fixed = 640 total (was 644)
          +1 mvnsite 1m 48s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 0s the patch passed
          +1 javadoc 1m 15s the patch passed
          +1 unit 1m 2s hadoop-hdfs-client in the patch passed.
          +1 unit 77m 35s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          108m 19s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-9038
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827080/HDFS-9038-011.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 3ecb99211ffb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 07650bc
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/16636/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16636/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16636/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16636/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 10 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 7m 7s trunk passed +1 compile 1m 35s trunk passed +1 checkstyle 0m 45s trunk passed +1 mvnsite 1m 34s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 19s trunk passed +1 javadoc 1m 18s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 21s the patch passed +1 compile 1m 26s the patch passed +1 cc 1m 26s the patch passed -1 javac 1m 26s hadoop-hdfs-project generated 1 new + 51 unchanged - 1 fixed = 52 total (was 52) -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 1 new + 639 unchanged - 5 fixed = 640 total (was 644) +1 mvnsite 1m 48s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 0s the patch passed +1 javadoc 1m 15s the patch passed +1 unit 1m 2s hadoop-hdfs-client in the patch passed. +1 unit 77m 35s hadoop-hdfs in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 108m 19s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-9038 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827080/HDFS-9038-011.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 3ecb99211ffb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 07650bc Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/16636/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16636/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16636/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16636/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          +1 lgtm. The javac warning is due to a deprecated function which we must invoke to handle wire compatibility. The checkstyle issue was also not introduced by this patch.

          Thanks again Brahma Reddy Battula for persistently driving this issue to closure.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited +1 lgtm. The javac warning is due to a deprecated function which we must invoke to handle wire compatibility. The checkstyle issue was also not introduced by this patch. Thanks again Brahma Reddy Battula for persistently driving this issue to closure.
          Hide
          vinayrpet Vinayakumar B added a comment -

          +1 for the latest changes.

          Thanks Brahma Reddy Battula, Arpit Agarwal, Chris Nauroth for the co-operations.

          Show
          vinayrpet Vinayakumar B added a comment - +1 for the latest changes. Thanks Brahma Reddy Battula , Arpit Agarwal , Chris Nauroth for the co-operations.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Pushed to 2.8.0. Hit numerous conflicts cherry-picking to branch-2.7 that I didn't look into. 2.7.4 will likely need an updated patch.

          Show
          arpitagarwal Arpit Agarwal added a comment - Pushed to 2.8.0. Hit numerous conflicts cherry-picking to branch-2.7 that I didn't look into. 2.7.4 will likely need an updated patch.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10401 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10401/)
          HDFS-9038. DFS reserved space is erroneously counted towards non-DFS (arp: rev 5f23abfa30ea29a5474513c463b4d462c0e824ee)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeCapacityReport.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/protocol/StorageReport.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10401 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10401/ ) HDFS-9038 . DFS reserved space is erroneously counted towards non-DFS (arp: rev 5f23abfa30ea29a5474513c463b4d462c0e824ee) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeCapacityReport.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/protocol/StorageReport.java
          Hide
          andrew.wang Andrew Wang added a comment -

          Gentle reminder, please set the appropriate 3.0.0 fix version when committing to trunk. Thanks!

          Show
          andrew.wang Andrew Wang added a comment - Gentle reminder, please set the appropriate 3.0.0 fix version when committing to trunk. Thanks!
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Uploading the branch-2.7 patch..And thanks for Arpit,Vinay and Chris for helping closure of this issue .Please review branch-2.7 patch.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Uploading the branch-2.7 patch..And thanks for Arpit,Vinay and Chris for helping closure of this issue .Please review branch-2.7 patch.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Reopening issue to run jenkins on branch-2.7 patch...

          Show
          brahmareddy Brahma Reddy Battula added a comment - Reopening issue to run jenkins on branch-2.7 patch...
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
          +1 mvninstall 6m 0s branch-2.7 passed
          +1 compile 1m 1s branch-2.7 passed with JDK v1.8.0_101
          +1 compile 0m 59s branch-2.7 passed with JDK v1.7.0_111
          +1 checkstyle 0m 30s branch-2.7 passed
          +1 mvnsite 1m 0s branch-2.7 passed
          +1 mvneclipse 0m 15s branch-2.7 passed
          +1 findbugs 2m 54s branch-2.7 passed
          +1 javadoc 0m 58s branch-2.7 passed with JDK v1.8.0_101
          +1 javadoc 1m 41s branch-2.7 passed with JDK v1.7.0_111
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 56s the patch passed with JDK v1.8.0_101
          +1 cc 0m 56s the patch passed
          -1 javac 0m 56s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_101 with JDK v1.8.0_101 generated 1 new + 44 unchanged - 1 fixed = 45 total (was 45)
          +1 compile 0m 59s the patch passed with JDK v1.7.0_111
          +1 cc 0m 59s the patch passed
          -1 javac 0m 59s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111 with JDK v1.7.0_111 generated 1 new + 47 unchanged - 1 fixed = 48 total (was 48)
          -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 523 unchanged - 3 fixed = 527 total (was 526)
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          -1 whitespace 0m 0s The patch has 7569 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 3m 11s The patch 137 line(s) with tabs.
          +1 findbugs 3m 8s the patch passed
          +1 javadoc 0m 55s the patch passed with JDK v1.8.0_101
          +1 javadoc 1m 40s the patch passed with JDK v1.7.0_111
          -1 unit 44m 22s hadoop-hdfs in the patch failed with JDK v1.7.0_111.
          -1 asflicense 0m 19s The patch generated 3 ASF License warnings.
          122m 57s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          JDK v1.7.0_111 Failed junit tests hadoop.hdfs.TestDatanodeRegistration
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Issue HDFS-9038
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827350/HDFS-9038-branch-2.7.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 4d52efe9b1e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / 67204f2
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/16656/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_101.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/16656/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16656/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16656/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16656/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16656/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16656/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16656/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16656/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. +1 mvninstall 6m 0s branch-2.7 passed +1 compile 1m 1s branch-2.7 passed with JDK v1.8.0_101 +1 compile 0m 59s branch-2.7 passed with JDK v1.7.0_111 +1 checkstyle 0m 30s branch-2.7 passed +1 mvnsite 1m 0s branch-2.7 passed +1 mvneclipse 0m 15s branch-2.7 passed +1 findbugs 2m 54s branch-2.7 passed +1 javadoc 0m 58s branch-2.7 passed with JDK v1.8.0_101 +1 javadoc 1m 41s branch-2.7 passed with JDK v1.7.0_111 +1 mvninstall 0m 51s the patch passed +1 compile 0m 56s the patch passed with JDK v1.8.0_101 +1 cc 0m 56s the patch passed -1 javac 0m 56s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_101 with JDK v1.8.0_101 generated 1 new + 44 unchanged - 1 fixed = 45 total (was 45) +1 compile 0m 59s the patch passed with JDK v1.7.0_111 +1 cc 0m 59s the patch passed -1 javac 0m 59s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111 with JDK v1.7.0_111 generated 1 new + 47 unchanged - 1 fixed = 48 total (was 48) -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 523 unchanged - 3 fixed = 527 total (was 526) +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch has 7569 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 3m 11s The patch 137 line(s) with tabs. +1 findbugs 3m 8s the patch passed +1 javadoc 0m 55s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 40s the patch passed with JDK v1.7.0_111 -1 unit 44m 22s hadoop-hdfs in the patch failed with JDK v1.7.0_111. -1 asflicense 0m 19s The patch generated 3 ASF License warnings. 122m 57s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.7.0_111 Failed junit tests hadoop.hdfs.TestDatanodeRegistration   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HDFS-9038 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827350/HDFS-9038-branch-2.7.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 4d52efe9b1e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 67204f2 Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/16656/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_101.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/16656/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16656/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16656/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16656/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16656/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16656/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16656/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16656/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          checkstyle : can be ignored ,those are related to file length,method parameters (>7) and hiddenfield.
          javac: which is not introduced by this patch.
          whitespace : Seems to be some problem with report..?
          asflicense : Not related to this patch
          unittest failures : Not related to this patch

          Arpit Agarwal can please review branch-2.7 patch..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - checkstyle : can be ignored ,those are related to file length,method parameters (>7) and hiddenfield. javac : which is not introduced by this patch. whitespace : Seems to be some problem with report..? asflicense : Not related to this patch unittest failures : Not related to this patch Arpit Agarwal can please review branch-2.7 patch..?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for the branch-2.7 patch also. Thanks Brahma.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for the branch-2.7 patch also. Thanks Brahma.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Pushed to branch-2.7.

          Show
          arpitagarwal Arpit Agarwal added a comment - Pushed to branch-2.7.
          Hide
          maobaolong maobaolong added a comment - - edited
          public long getNonDfsUsed() throws IOException {
              long actualNonDfsUsed = getActualNonDfsUsed();
              if (actualNonDfsUsed < reserved) {
                return 0L;
              }
              return actualNonDfsUsed - reserved;
            }
          

          The code block above is the function to caculate nonDfsUsed, but in fact it will let the result to be 0L out of expect. Such as this following situation:

          du.reserved = 50G
          Disk Capacity = 2048G
          Disk Available = 2000G
          Dfs used = 30G

          usage.getUsed() = dirFile.getTotalSpace() - dirFile.getFreeSpace()
          = 2048G - 2000G
          = 48G
          getActualNonDfsUsed = usage.getUsed() - getDfsUsed()
          = 48G - 30G
          = 18G
          18G < 50G, so the function `getNonDfsUsed` actualNonDfsUsed < reserved, and the NonDfsUsed will return 0, is that logic make sense?

          Show
          maobaolong maobaolong added a comment - - edited public long getNonDfsUsed() throws IOException { long actualNonDfsUsed = getActualNonDfsUsed(); if (actualNonDfsUsed < reserved) { return 0L; } return actualNonDfsUsed - reserved; } The code block above is the function to caculate nonDfsUsed, but in fact it will let the result to be 0L out of expect. Such as this following situation: du.reserved = 50G Disk Capacity = 2048G Disk Available = 2000G Dfs used = 30G usage.getUsed() = dirFile.getTotalSpace() - dirFile.getFreeSpace() = 2048G - 2000G = 48G getActualNonDfsUsed = usage.getUsed() - getDfsUsed() = 48G - 30G = 18G 18G < 50G, so the function `getNonDfsUsed` actualNonDfsUsed < reserved, and the NonDfsUsed will return 0, is that logic make sense?
          Hide
          vinayrpet Vinayakumar B added a comment -
            /**
             * Unplanned Non-DFS usage, i.e. Extra usage beyond reserved.
             *
             * @return
             * @throws IOException
             */
            public long getNonDfsUsed() throws IOException {
              long actualNonDfsUsed = getActualNonDfsUsed();
              if (actualNonDfsUsed < reserved) {
                return 0L;
              }
              return actualNonDfsUsed - reserved;
            }

          As the javadoc says, its the UnPlanned Non-DFS usage beyond reserved. If the actual NonDFS usage is within reserved limits, this returns 0. You can see multiple discussions happened earlier in comments.
          So YES, this logic makes sense.

          Show
          vinayrpet Vinayakumar B added a comment - /** * Unplanned Non-DFS usage, i.e. Extra usage beyond reserved. * * @ return * @ throws IOException */ public long getNonDfsUsed() throws IOException { long actualNonDfsUsed = getActualNonDfsUsed(); if (actualNonDfsUsed < reserved) { return 0L; } return actualNonDfsUsed - reserved; } As the javadoc says, its the UnPlanned Non-DFS usage beyond reserved. If the actual NonDFS usage is within reserved limits, this returns 0. You can see multiple discussions happened earlier in comments. So YES, this logic makes sense.
          Hide
          maobaolong maobaolong added a comment -

          Shreyas Vinayakumar Thank you, I have known the NonDFS used mean.

          Show
          maobaolong maobaolong added a comment - Shreyas Vinayakumar Thank you, I have known the NonDFS used mean.

            People

            • Assignee:
              brahmareddy Brahma Reddy Battula
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development