Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9653

Expose the number of blocks pending deletion through dfsadmin report command

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: hdfs-client, tools
    • Labels:
      None
    • Target Version/s:

      Description

      HDFS-5986 adds Number of Blocks Pending Deletion on namenode UI and JMX, propose to expose this from hdfs dfsadmin -report as well. This is useful when hadoop admin was not able to access UI (e.g on cloud), he/she can directly use command to retrieve this information.

      1. HDFS-9653.001.patch
        9 kB
        Weiwei Yang
      2. HDFS-9653.002.patch
        13 kB
        Weiwei Yang

        Activity

        Hide
        cheersyang Weiwei Yang added a comment -

        Submitted patch, it is a pretty straight forward change, please kindly help to review. Thanks

        Show
        cheersyang Weiwei Yang added a comment - Submitted patch, it is a pretty straight forward change, please kindly help to review. Thanks
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 28s Maven dependency ordering for branch
        +1 mvninstall 7m 32s trunk passed
        +1 compile 1m 14s trunk passed with JDK v1.8.0_66
        +1 compile 1m 20s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 1m 26s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 48s trunk passed
        +1 javadoc 1m 28s trunk passed with JDK v1.8.0_66
        +1 javadoc 2m 13s trunk passed with JDK v1.7.0_91
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 16s the patch passed
        +1 compile 1m 16s the patch passed with JDK v1.8.0_66
        +1 cc 1m 16s the patch passed
        +1 javac 1m 16s the patch passed
        +1 compile 1m 22s the patch passed with JDK v1.7.0_91
        +1 cc 1m 22s the patch passed
        +1 javac 1m 22s the patch passed
        +1 checkstyle 0m 29s the patch passed
        +1 mvnsite 1m 21s the patch passed
        +1 mvneclipse 0m 21s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 4m 14s the patch passed
        +1 javadoc 1m 24s the patch passed with JDK v1.8.0_66
        +1 javadoc 2m 5s the patch passed with JDK v1.7.0_91
        +1 unit 0m 47s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        -1 unit 51m 52s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        +1 unit 51m 41s hadoop-hdfs in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 24s Patch does not generate ASF License warnings.
        142m 36s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.server.datanode.TestBlockScanner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782833/HDFS-9653.001.patch
        JIRA Issue HDFS-9653
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 9774ae0201e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3fe5728
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14146/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14146/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14146/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Max memory used 77MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14146/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 28s Maven dependency ordering for branch +1 mvninstall 7m 32s trunk passed +1 compile 1m 14s trunk passed with JDK v1.8.0_66 +1 compile 1m 20s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 48s trunk passed +1 javadoc 1m 28s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 13s trunk passed with JDK v1.7.0_91 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 16s the patch passed +1 compile 1m 16s the patch passed with JDK v1.8.0_66 +1 cc 1m 16s the patch passed +1 javac 1m 16s the patch passed +1 compile 1m 22s the patch passed with JDK v1.7.0_91 +1 cc 1m 22s the patch passed +1 javac 1m 22s the patch passed +1 checkstyle 0m 29s the patch passed +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 14s the patch passed +1 javadoc 1m 24s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 5s the patch passed with JDK v1.7.0_91 +1 unit 0m 47s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 51m 52s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. +1 unit 51m 41s hadoop-hdfs in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 142m 36s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.datanode.TestBlockScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782833/HDFS-9653.001.patch JIRA Issue HDFS-9653 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 9774ae0201e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3fe5728 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14146/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14146/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14146/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14146/console This message was automatically generated.
        Hide
        cheersyang Weiwei Yang added a comment -

        The UT failure seems to be irrelevant to this patch. These 3 tests can run successfully with JDK 1.8.0_66 on my local environment. hadoop.hdfs.TestDFSUpgradeFromImage seems to be HDFS-9476.

        Show
        cheersyang Weiwei Yang added a comment - The UT failure seems to be irrelevant to this patch. These 3 tests can run successfully with JDK 1.8.0_66 on my local environment. hadoop.hdfs.TestDFSUpgradeFromImage seems to be HDFS-9476 .
        Hide
        eyang Eric Yang added a comment -

        Can you generate one more patch for branch-2? Assuming that you want this in 2.x as well.

        Show
        eyang Eric Yang added a comment - Can you generate one more patch for branch-2? Assuming that you want this in 2.x as well.
        Hide
        cheersyang Weiwei Yang added a comment -

        Submit a patch for branch-2

        Show
        cheersyang Weiwei Yang added a comment - Submit a patch for branch-2
        Hide
        cheersyang Weiwei Yang added a comment -

        Thanks Eric. I just submitted a patch for branch-2.

        Show
        cheersyang Weiwei Yang added a comment - Thanks Eric. I just submitted a patch for branch-2.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 1m 52s Maven dependency ordering for branch
        +1 mvninstall 7m 13s branch-2 passed
        +1 compile 1m 53s branch-2 passed with JDK v1.8.0_66
        +1 compile 1m 38s branch-2 passed with JDK v1.7.0_91
        +1 checkstyle 0m 30s branch-2 passed
        +1 mvnsite 1m 30s branch-2 passed
        +1 mvneclipse 0m 29s branch-2 passed
        -1 findbugs 2m 1s hadoop-hdfs-project/hadoop-hdfs-client in branch-2 has 5 extant Findbugs warnings.
        +1 javadoc 1m 54s branch-2 passed with JDK v1.8.0_66
        +1 javadoc 2m 18s branch-2 passed with JDK v1.7.0_91
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 30s the patch passed
        +1 compile 1m 51s the patch passed with JDK v1.8.0_66
        +1 cc 1m 51s the patch passed
        +1 javac 1m 51s the patch passed
        +1 compile 1m 27s the patch passed with JDK v1.7.0_91
        +1 cc 1m 27s the patch passed
        +1 javac 1m 27s the patch passed
        +1 checkstyle 0m 27s the patch passed
        +1 mvnsite 1m 26s the patch passed
        +1 mvneclipse 0m 21s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 4m 13s the patch passed
        +1 javadoc 1m 37s the patch passed with JDK v1.8.0_66
        +1 javadoc 2m 17s the patch passed with JDK v1.7.0_91
        +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        -1 unit 62m 42s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        +1 unit 1m 15s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        -1 unit 66m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        173m 25s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
          hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.TestBlockStoragePolicy
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAAppend
          hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          hadoop.hdfs.server.datanode.TestBlockScanner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:babe025
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783019/HDFS-9653-branch-2.001.patch
        JIRA Issue HDFS-9653
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux c71313657e7e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / bb2a9a7
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14156/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14156/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14156/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14156/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14156/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14156/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Max memory used 77MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14156/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 52s Maven dependency ordering for branch +1 mvninstall 7m 13s branch-2 passed +1 compile 1m 53s branch-2 passed with JDK v1.8.0_66 +1 compile 1m 38s branch-2 passed with JDK v1.7.0_91 +1 checkstyle 0m 30s branch-2 passed +1 mvnsite 1m 30s branch-2 passed +1 mvneclipse 0m 29s branch-2 passed -1 findbugs 2m 1s hadoop-hdfs-project/hadoop-hdfs-client in branch-2 has 5 extant Findbugs warnings. +1 javadoc 1m 54s branch-2 passed with JDK v1.8.0_66 +1 javadoc 2m 18s branch-2 passed with JDK v1.7.0_91 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 30s the patch passed +1 compile 1m 51s the patch passed with JDK v1.8.0_66 +1 cc 1m 51s the patch passed +1 javac 1m 51s the patch passed +1 compile 1m 27s the patch passed with JDK v1.7.0_91 +1 cc 1m 27s the patch passed +1 javac 1m 27s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 1m 26s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 13s the patch passed +1 javadoc 1m 37s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 17s the patch passed with JDK v1.7.0_91 +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 62m 42s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 1m 15s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 66m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 173m 25s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestBlockStoragePolicy JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.datanode.TestBlockScanner Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783019/HDFS-9653-branch-2.001.patch JIRA Issue HDFS-9653 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux c71313657e7e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / bb2a9a7 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14156/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/14156/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14156/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14156/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14156/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14156/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14156/console This message was automatically generated.
        Hide
        cheersyang Weiwei Yang added a comment -

        The findbugs warnings are not caused by this patch.

        Show
        cheersyang Weiwei Yang added a comment - The findbugs warnings are not caused by this patch.
        Hide
        cheersyang Weiwei Yang added a comment -

        I have run failed UT (branch-2) on my local env, TestBlockScanner,TestHAAppend,TestRBWBlockInvalidation,TestDataNodeHotSwapVolumes, they all succeed. They should not be caused by this patch.

        Show
        cheersyang Weiwei Yang added a comment - I have run failed UT (branch-2) on my local env, TestBlockScanner,TestHAAppend,TestRBWBlockInvalidation,TestDataNodeHotSwapVolumes, they all succeed. They should not be caused by this patch.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        If the clients are upgraded and the namenode is not upgraded and if we issue 'hdfs dfsadmin -report' command, then what will be the default value of Pending Deletion blocks ?

        Show
        shahrs87 Rushabh S Shah added a comment - If the clients are upgraded and the namenode is not upgraded and if we issue 'hdfs dfsadmin -report' command, then what will be the default value of Pending Deletion blocks ?
        Hide
        cheersyang Weiwei Yang added a comment -

        Hello Rushabh Shah

        I think that's gonna fail. This patch changes the protocol between client and namenode by asking one more information from status call, if namenode is not upgraded, that piece is missing and right now namenode sets fixed size of array and client retrieve by index, so .. it will be an out of index. This can be fixed from client side by adding a check before reading the status returned by namenode. This is not the problem this patch intents to fix.

        Show
        cheersyang Weiwei Yang added a comment - Hello Rushabh Shah I think that's gonna fail. This patch changes the protocol between client and namenode by asking one more information from status call, if namenode is not upgraded, that piece is missing and right now namenode sets fixed size of array and client retrieve by index, so .. it will be an out of index. This can be fixed from client side by adding a check before reading the status returned by namenode. This is not the problem this patch intents to fix.
        Hide
        eyang Eric Yang added a comment -

        WeiWei,

        In rolling upgrade, the new client may talk to old version of name node. If we can handle the missing data gracefully on the client side, it would be better.

        Show
        eyang Eric Yang added a comment - WeiWei, In rolling upgrade, the new client may talk to old version of name node. If we can handle the missing data gracefully on the client side, it would be better.
        Hide
        cheersyang Weiwei Yang added a comment -

        According to Eric and Rushabh's comments, submitted a new patch. This patch handles the potential different version between hdfs-client and namenode.

        Show
        cheersyang Weiwei Yang added a comment - According to Eric and Rushabh's comments, submitted a new patch. This patch handles the potential different version between hdfs-client and namenode.
        Hide
        cheersyang Weiwei Yang added a comment -

        Hello Eric Yang

        I updated the patch to handle that situation, and I have added a test to cover potential client protocol compatibility issue. I added a check on client side before retrieving states by index, and return -1 if it goes out of array (-1 is also the default value set on server side). This patch can be applied to both branc-2 and branch-2.8 too, I have tested that. Please help to review the 002 patch. Thanks a lot.

        Show
        cheersyang Weiwei Yang added a comment - Hello Eric Yang I updated the patch to handle that situation, and I have added a test to cover potential client protocol compatibility issue. I added a check on client side before retrieving states by index, and return -1 if it goes out of array (-1 is also the default value set on server side). This patch can be applied to both branc-2 and branch-2.8 too, I have tested that. Please help to review the 002 patch. Thanks a lot.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 15s Maven dependency ordering for branch
        +1 mvninstall 7m 25s trunk passed
        +1 compile 1m 31s trunk passed with JDK v1.8.0_66
        +1 compile 1m 22s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 26s trunk passed
        +1 mvnsite 1m 24s trunk passed
        +1 mvneclipse 0m 24s trunk passed
        +1 findbugs 3m 46s trunk passed
        +1 javadoc 1m 29s trunk passed with JDK v1.8.0_66
        +1 javadoc 2m 12s trunk passed with JDK v1.7.0_91
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 17s the patch passed
        +1 compile 1m 28s the patch passed with JDK v1.8.0_66
        +1 cc 1m 28s the patch passed
        +1 javac 1m 28s the patch passed
        +1 compile 1m 22s the patch passed with JDK v1.7.0_91
        +1 cc 1m 22s the patch passed
        +1 javac 1m 22s the patch passed
        +1 checkstyle 0m 26s hadoop-hdfs-project: patch generated 0 new + 555 unchanged - 1 fixed = 555 total (was 556)
        +1 mvnsite 1m 23s the patch passed
        +1 mvneclipse 0m 20s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 4m 8s the patch passed
        +1 javadoc 1m 28s the patch passed with JDK v1.8.0_66
        +1 javadoc 2m 5s the patch passed with JDK v1.7.0_91
        +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        -1 unit 61m 13s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        -1 unit 59m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        160m 5s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestErasureCodeBenchmarkThroughput
          hadoop.hdfs.server.datanode.TestBlockScanner
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestStartup



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784012/HDFS-9653.002.patch
        JIRA Issue HDFS-9653
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 1ad4d87feb13 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 99829eb
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14227/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14227/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14227/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14227/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14227/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Max memory used 77MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14227/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 7m 25s trunk passed +1 compile 1m 31s trunk passed with JDK v1.8.0_66 +1 compile 1m 22s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 26s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 3m 46s trunk passed +1 javadoc 1m 29s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 12s trunk passed with JDK v1.7.0_91 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 17s the patch passed +1 compile 1m 28s the patch passed with JDK v1.8.0_66 +1 cc 1m 28s the patch passed +1 javac 1m 28s the patch passed +1 compile 1m 22s the patch passed with JDK v1.7.0_91 +1 cc 1m 22s the patch passed +1 javac 1m 22s the patch passed +1 checkstyle 0m 26s hadoop-hdfs-project: patch generated 0 new + 555 unchanged - 1 fixed = 555 total (was 556) +1 mvnsite 1m 23s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 8s the patch passed +1 javadoc 1m 28s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 5s the patch passed with JDK v1.7.0_91 +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 61m 13s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 59m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 160m 5s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestErasureCodeBenchmarkThroughput   hadoop.hdfs.server.datanode.TestBlockScanner JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestStartup Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784012/HDFS-9653.002.patch JIRA Issue HDFS-9653 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 1ad4d87feb13 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 99829eb Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14227/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14227/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14227/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14227/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14227/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14227/console This message was automatically generated.
        Hide
        cheersyang Weiwei Yang added a comment -

        The UT failures are not related. I have tested and run them locally and they passed. There are existing JIRA opening to track failures, see HDFS-9660.

        Show
        cheersyang Weiwei Yang added a comment - The UT failures are not related. I have tested and run them locally and they passed. There are existing JIRA opening to track failures, see HDFS-9660 .
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9174 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9174/)
        HDFS-9653. Added blocks pending deletion report to dfsadmin. (Weiwei (eyang: rev 10a2bc0dffaece216eb9a6bac3236a086b9ece31)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9174 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9174/ ) HDFS-9653 . Added blocks pending deletion report to dfsadmin. (Weiwei (eyang: rev 10a2bc0dffaece216eb9a6bac3236a086b9ece31) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
        Hide
        cheersyang Weiwei Yang added a comment -

        Thanks a lot Eric Yang

        Show
        cheersyang Weiwei Yang added a comment - Thanks a lot Eric Yang

          People

          • Assignee:
            cheersyang Weiwei Yang
            Reporter:
            cheersyang Weiwei Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development