Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9600

do not check replication if the block is under construction

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None

      Description

      When appending a file, we will update pipeline to bump a new GS and the old GS will be considered as out of date. When changing GS, in BlockInfo.setGenerationStampAndVerifyReplicas we will remove replicas having old GS which means we will remove all replicas because no DN has new GS until the block with new GS is added to blockMaps again by DatanodeProtocol.blockReceivedAndDeleted.

      If we check replication of this block before it is added back, it will be regarded as missing. The probability is low but if there are decommissioning nodes the DecommissionManager.Monitor will scan all blocks belongs to decommissioning nodes with a very fast speed so the probability of finding missing block is very high but actually they are not missing.

      Furthermore, after closing the appended file, in FSNamesystem.finalizeINodeFileUnderConstruction, it will checkReplication. If some of nodes are decommissioning, this block with new GS will be added to UnderReplicatedBlocks map so there are two blocks with same ID in this map, one is in QUEUE_WITH_CORRUPT_BLOCKS and the other is in QUEUE_HIGHEST_PRIORITY or QUEUE_UNDER_REPLICATED. And there will be many missing blocks warning in NameNode website but there is no corrupt files...

      Therefore, I think the solution is we should not check replication if the block is under construction. We only check complete blocks.

      1. HDFS-9600-branch-2.6.patch
        6 kB
        Phil Yang
      2. HDFS-9600-branch-2.7.patch
        7 kB
        Phil Yang
      3. HDFS-9600-branch-2.patch
        7 kB
        Phil Yang
      4. HDFS-9600-v4.patch
        7 kB
        Phil Yang
      5. HDFS-9600-v3.patch
        7 kB
        Phil Yang
      6. HDFS-9600-v2.patch
        5 kB
        Phil Yang
      7. HDFS-9600-v1.patch
        0.9 kB
        Phil Yang

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Thanks for looking into and commenting Andrew!

          I agree that how the status is set is confusing here.

          For open files, we only let it decom if the UC block stays above minReplication (default 1). Note the curReplicas > minReplication check.

          The isReplicationInProgress method initialize status to false, for replicationFactor==1 case, the check curReplicas > minReplication will be false, thus the method will return false, and the node that contains the single replica will be marked as decommissioned, thus the block will be lost , unless we over replicate all replicas on the node before decommissioning for replicationFactor==1 case. Am I right? Did we do over-replicate before decommissioning?

          One point in my earlier comment is, If a replica is currently being written to (under construction), in order for us to decommission this node, either 1, the client need to reconstruct the write pipeline without this node, or 2, we need to wait for the block to be complete, and satisfy the replication count before this node can be decommissioned. Since it may take long to finish writing a block, I think likely it's case 1. If we go with case 1, then for replicationFactor==1 case, the client need to rewrite all data of the same block to other node. I wonder if this is the current behavior?

          Hope other folks who are more familiar with the code can answer your question. It'd be nice to have some good document about the handling.

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Thanks for looking into and commenting Andrew! I agree that how the status is set is confusing here. For open files, we only let it decom if the UC block stays above minReplication (default 1). Note the curReplicas > minReplication check. The isReplicationInProgress method initialize status to false, for replicationFactor==1 case, the check curReplicas > minReplication will be false, thus the method will return false, and the node that contains the single replica will be marked as decommissioned, thus the block will be lost , unless we over replicate all replicas on the node before decommissioning for replicationFactor==1 case. Am I right? Did we do over-replicate before decommissioning? One point in my earlier comment is, If a replica is currently being written to (under construction), in order for us to decommission this node, either 1, the client need to reconstruct the write pipeline without this node, or 2, we need to wait for the block to be complete, and satisfy the replication count before this node can be decommissioned. Since it may take long to finish writing a block, I think likely it's case 1. If we go with case 1, then for replicationFactor==1 case, the client need to rewrite all data of the same block to other node. I wonder if this is the current behavior? Hope other folks who are more familiar with the code can answer your question. It'd be nice to have some good document about the handling. Thanks.
          Hide
          andrew.wang Andrew Wang added a comment -

          1. If the block is not complete, especially if the block is being written to right now, we seem to be able to still decommission this node (isReplicationInProgress returns false). That may be ok for replication factor bigger than 1 (let the remaining replicas to carry on the ongoing write), but if it's 1, then we would lost the replica, and the block. Isn't that a problem?

          For open files, we only let it decom if the UC block stays above minReplication (default 1). Note the curReplicas > minReplication check.

          2. If the block is complete, and the replication factor is 1, similarly, the isReplicationInProgress method will return false and we are still able to decommission the node.

          Note that the liveReplicas from the countNodes function does not include decommissioning nodes. So for this case curReplicas will be 0, less than the curExpectedReplicas of 1. The way status is set is pretty confusing in this function.

          One question I have is where we add blocks to neededReplications at the end of isReplicationInProgress. We should be checking that the block is complete here right?

          Show
          andrew.wang Andrew Wang added a comment - 1. If the block is not complete, especially if the block is being written to right now, we seem to be able to still decommission this node (isReplicationInProgress returns false). That may be ok for replication factor bigger than 1 (let the remaining replicas to carry on the ongoing write), but if it's 1, then we would lost the replica, and the block. Isn't that a problem? For open files, we only let it decom if the UC block stays above minReplication (default 1). Note the curReplicas > minReplication check. 2. If the block is complete, and the replication factor is 1, similarly, the isReplicationInProgress method will return false and we are still able to decommission the node. Note that the liveReplicas from the countNodes function does not include decommissioning nodes. So for this case curReplicas will be 0, less than the curExpectedReplicas of 1. The way status is set is pretty confusing in this function. One question I have is where we add blocks to neededReplications at the end of isReplicationInProgress . We should be checking that the block is complete here right?
          Hide
          yzhangal Yongjun Zhang added a comment -

          Thanks Phil Yang!

          I studied the code further and have the following thoughts:

           /**
             * Return true if there are any blocks on this node that have not
             * yet reached their replication factor. Otherwise returns false.
             */
            boolean isReplicationInProgress(DatanodeDescriptor srcNode) {
          

          As you said, this method is used by DatanodeManager.checkDecommissionState, if all blocks hold by a decommissioning node are not "isReplicationInProgress", then it will be set decommissioned. The current criteria is:

                  if (curReplicas < curExpectedReplicas
                      || !isPlacementPolicySatisfied(block)) {
          

          We do not check whether the block is complete or not, because the most concern is whether we have enough replicas. However, I found couple of things here:

          1. If the block is not complete, especially if the block is being written to right now, we seem to be able to still decommission this node (isReplicationInProgress returns false). That may be ok for replication factor bigger than 1 (let the remaining replicas to carry on the ongoing write), but if it's 1, then we would lost the replica, and the block. Isn't that a problem?

          2. If the block is complete, and the replication factor is 1, similarly, the isReplicationInProgress method will return false and we are still able to decommission the node.

          So seems that we need to handle replication factor 1 with extra care. The solution would be to over-replicate first, then decommission the node. I am not sure whether the de-commissioner handles that.

          Thoughts?

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Thanks Phil Yang ! I studied the code further and have the following thoughts: /** * Return true if there are any blocks on this node that have not * yet reached their replication factor. Otherwise returns false . */ boolean isReplicationInProgress(DatanodeDescriptor srcNode) { As you said, this method is used by DatanodeManager.checkDecommissionState, if all blocks hold by a decommissioning node are not "isReplicationInProgress", then it will be set decommissioned. The current criteria is: if (curReplicas < curExpectedReplicas || !isPlacementPolicySatisfied(block)) { We do not check whether the block is complete or not, because the most concern is whether we have enough replicas. However, I found couple of things here: 1. If the block is not complete, especially if the block is being written to right now, we seem to be able to still decommission this node (isReplicationInProgress returns false). That may be ok for replication factor bigger than 1 (let the remaining replicas to carry on the ongoing write), but if it's 1, then we would lost the replica, and the block. Isn't that a problem? 2. If the block is complete, and the replication factor is 1, similarly, the isReplicationInProgress method will return false and we are still able to decommission the node. So seems that we need to handle replication factor 1 with extra care. The solution would be to over-replicate first, then decommission the node. I am not sure whether the de-commissioner handles that. Thoughts? Thanks.
          Hide
          yangzhe1991 Phil Yang added a comment -

          Hi Yongjun Zhang
          BlockManager.isReplicationInProgress is used by DatanodeManager.checkDecommissionState, if all blocks hold by a decommissioning node are not "isReplicationInProgress", it will be set as decommissioned. If there is a block that is not complete, it will not be NeededReplication but if it is not isReplicationInProgress, the datanode will be set as decommissioned and the block may be missed.

          Show
          yangzhe1991 Phil Yang added a comment - Hi Yongjun Zhang BlockManager.isReplicationInProgress is used by DatanodeManager.checkDecommissionState, if all blocks hold by a decommissioning node are not "isReplicationInProgress", it will be set as decommissioned. If there is a block that is not complete, it will not be NeededReplication but if it is not isReplicationInProgress, the datanode will be set as decommissioned and the block may be missed.
          Hide
          yzhangal Yongjun Zhang added a comment -

          HI Phil Yang, Tsz Wo Nicholas Sze and Vinayakumar B,

          Thanks for your earlier work here. I noticed that in branch-2.6

            /**
             * Return true if there are any blocks on this node that have not
             * yet reached their replication factor. Otherwise returns false.
             */
            boolean isReplicationInProgress(DatanodeDescriptor srcNode) {
              boolean status = false;
              boolean firstReplicationLog = true;
              int underReplicatedBlocks = 0;
              int decommissionOnlyReplicas = 0;
              int underReplicatedInOpenFiles = 0;
              final Iterator<? extends Block> it = srcNode.getBlockIterator();
              while(it.hasNext()) {
                final Block block = it.next();
                BlockCollection bc = blocksMap.getBlockCollection(block);
          
                if (bc != null) {
                  NumberReplicas num = countNodes(block);
                  int curReplicas = num.liveReplicas();
                  int curExpectedReplicas = getReplication(block);
          
                  if (curReplicas < curExpectedReplicas
                      || !isPlacementPolicySatisfied(block)) {
          

          And clause {{ blockInfo.isComplete()}} is added to the following method but not in the above method, Would any of you please explain why blockInfo.isComplete() is not needed in the above code? Thanks a lot.

            boolean isNeededReplication(Block b, int expected, int current) {
              BlockInfo blockInfo;
              if (b instanceof BlockInfo) {
                blockInfo = (BlockInfo) b;
              } else {
                blockInfo = getStoredBlock(b);
              }
              return blockInfo.isComplete()
                  && (current < expected || !isPlacementPolicySatisfied(b));
            }
          
          Show
          yzhangal Yongjun Zhang added a comment - HI Phil Yang , Tsz Wo Nicholas Sze and Vinayakumar B , Thanks for your earlier work here. I noticed that in branch-2.6 /** * Return true if there are any blocks on this node that have not * yet reached their replication factor. Otherwise returns false . */ boolean isReplicationInProgress(DatanodeDescriptor srcNode) { boolean status = false ; boolean firstReplicationLog = true ; int underReplicatedBlocks = 0; int decommissionOnlyReplicas = 0; int underReplicatedInOpenFiles = 0; final Iterator<? extends Block> it = srcNode.getBlockIterator(); while (it.hasNext()) { final Block block = it.next(); BlockCollection bc = blocksMap.getBlockCollection(block); if (bc != null ) { NumberReplicas num = countNodes(block); int curReplicas = num.liveReplicas(); int curExpectedReplicas = getReplication(block); if (curReplicas < curExpectedReplicas || !isPlacementPolicySatisfied(block)) { And clause {{ blockInfo.isComplete()}} is added to the following method but not in the above method, Would any of you please explain why blockInfo.isComplete() is not needed in the above code? Thanks a lot. boolean isNeededReplication(Block b, int expected, int current) { BlockInfo blockInfo; if (b instanceof BlockInfo) { blockInfo = (BlockInfo) b; } else { blockInfo = getStoredBlock(b); } return blockInfo.isComplete() && (current < expected || !isPlacementPolicySatisfied(b)); }
          Hide
          yangzhe1991 Phil Yang added a comment -

          Thanks

          Show
          yangzhe1991 Phil Yang added a comment - Thanks
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9064 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9064/)
          HDFS-9600. do not check replication if the block is under construction (vinayakumarb: rev 34cd7cd76505d01ec251e30837c94ab03319a0c1)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9064 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9064/ ) HDFS-9600 . do not check replication if the block is under construction (vinayakumarb: rev 34cd7cd76505d01ec251e30837c94ab03319a0c1) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          vinayrpet Vinayakumar B added a comment -

          Merged to branch-2.8 as well.

          Show
          vinayrpet Vinayakumar B added a comment - Merged to branch-2.8 as well.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Committed to trunk, branch-2, branch-2.7 and branch-2.6

          Thanks all.

          Show
          vinayrpet Vinayakumar B added a comment - Committed to trunk, branch-2, branch-2.7 and branch-2.6 Thanks all.
          Hide
          vinayrpet Vinayakumar B added a comment -

          The native build fails when libwebhdfs in contrib is built. This is not the case if you simply do -Pnative. I think it is HDFS-8346.

          Might be another reason for failure in branch-2.
          But I have seen with both docker and non-docker mode in branch-2.6. It fails in branch-2.6 with docker mode because dev-support/DockerFile doesnot exist in branch-2.6.

          Show
          vinayrpet Vinayakumar B added a comment - The native build fails when libwebhdfs in contrib is built. This is not the case if you simply do -Pnative. I think it is HDFS-8346 . Might be another reason for failure in branch-2. But I have seen with both docker and non-docker mode in branch-2.6. It fails in branch-2.6 with docker mode because dev-support/DockerFile doesnot exist in branch-2.6.
          Hide
          kihwal Kihwal Lee added a comment -

          I think, its the problem with docker image used to run branch-2.6. In Non-docker mode, its fine.

          The native build fails when libwebhdfs in contrib is built. This is not the case if you simply do -Pnative. I think it is HDFS-8346.

          Show
          kihwal Kihwal Lee added a comment - I think, its the problem with docker image used to run branch-2.6. In Non-docker mode, its fine. The native build fails when libwebhdfs in contrib is built. This is not the case if you simply do -Pnative . I think it is HDFS-8346 .
          Hide
          yangzhe1991 Phil Yang added a comment -

          Ok, what should I do now?

          Show
          yangzhe1991 Phil Yang added a comment - Ok, what should I do now?
          Hide
          vinayrpet Vinayakumar B added a comment -

          It failed on native code?

          I think, its the problem with docker image used to run branch-2.6
          In Non-docker mode, its fine.

          Show
          vinayrpet Vinayakumar B added a comment - It failed on native code? I think, its the problem with docker image used to run branch-2.6 In Non-docker mode, its fine.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 10m 28s branch-2.6 passed
          -1 compile 0m 45s hadoop-hdfs in branch-2.6 failed with JDK v1.8.0_66.
          -1 compile 0m 45s hadoop-hdfs in branch-2.6 failed with JDK v1.7.0_91.
          +1 checkstyle 0m 21s branch-2.6 passed
          +1 mvnsite 0m 59s branch-2.6 passed
          +1 mvneclipse 0m 20s branch-2.6 passed
          -1 findbugs 2m 55s hadoop-hdfs-project/hadoop-hdfs in branch-2.6 has 272 extant Findbugs warnings.
          +1 javadoc 1m 12s branch-2.6 passed with JDK v1.8.0_66
          +1 javadoc 1m 53s branch-2.6 passed with JDK v1.7.0_91
          +1 mvninstall 0m 55s the patch passed
          -1 compile 0m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 javac 0m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 compile 0m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 javac 0m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 checkstyle 0m 17s the patch passed
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 1s The patch has 2997 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 1m 14s The patch has 95 line(s) with tabs.
          +1 findbugs 3m 13s the patch passed
          +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 53s the patch passed with JDK v1.7.0_91
          -1 unit 0m 48s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 0m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 37s Patch generated 73 ASF License warnings.
          48m 17s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:date2015-12-30
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779934/HDFS-9600-branch-2.6.patch
          JIRA Issue HDFS-9600
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 34689ff9cf1c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.6 / 0f9dd48
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          findbugs v1.3.9
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14013/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 70MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14013/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 10m 28s branch-2.6 passed -1 compile 0m 45s hadoop-hdfs in branch-2.6 failed with JDK v1.8.0_66. -1 compile 0m 45s hadoop-hdfs in branch-2.6 failed with JDK v1.7.0_91. +1 checkstyle 0m 21s branch-2.6 passed +1 mvnsite 0m 59s branch-2.6 passed +1 mvneclipse 0m 20s branch-2.6 passed -1 findbugs 2m 55s hadoop-hdfs-project/hadoop-hdfs in branch-2.6 has 272 extant Findbugs warnings. +1 javadoc 1m 12s branch-2.6 passed with JDK v1.8.0_66 +1 javadoc 1m 53s branch-2.6 passed with JDK v1.7.0_91 +1 mvninstall 0m 55s the patch passed -1 compile 0m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 javac 0m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 compile 0m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 javac 0m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 1s The patch has 2997 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 1m 14s The patch has 95 line(s) with tabs. +1 findbugs 3m 13s the patch passed +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 53s the patch passed with JDK v1.7.0_91 -1 unit 0m 48s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 0m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 37s Patch generated 73 ASF License warnings. 48m 17s Subsystem Report/Notes Docker Image:yetus/hadoop:date2015-12-30 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779934/HDFS-9600-branch-2.6.patch JIRA Issue HDFS-9600 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 34689ff9cf1c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.6 / 0f9dd48 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 compile https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt findbugs v1.3.9 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html compile https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14013/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14013/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 70MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14013/console This message was automatically generated.
          Hide
          yangzhe1991 Phil Yang added a comment -

          Why 2.6 patch failed on compiling...
          It failed on native code?

          Show
          yangzhe1991 Phil Yang added a comment - Why 2.6 patch failed on compiling... It failed on native code?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 10m 32s branch-2.6 passed
          -1 compile 0m 42s hadoop-hdfs in branch-2.6 failed with JDK v1.8.0_66.
          -1 compile 0m 44s hadoop-hdfs in branch-2.6 failed with JDK v1.7.0_91.
          +1 checkstyle 0m 21s branch-2.6 passed
          +1 mvnsite 0m 57s branch-2.6 passed
          +1 mvneclipse 0m 20s branch-2.6 passed
          -1 findbugs 2m 54s hadoop-hdfs-project/hadoop-hdfs in branch-2.6 has 272 extant Findbugs warnings.
          +1 javadoc 1m 11s branch-2.6 passed with JDK v1.8.0_66
          +1 javadoc 1m 52s branch-2.6 passed with JDK v1.7.0_91
          +1 mvninstall 0m 53s the patch passed
          -1 compile 0m 39s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 javac 0m 39s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 compile 0m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 javac 0m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 checkstyle 0m 17s the patch passed
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          -1 whitespace 0m 1s The patch has 2664 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 1m 10s The patch has 95 line(s) with tabs.
          +1 findbugs 2m 52s the patch passed
          +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 48s the patch passed with JDK v1.7.0_91
          -1 unit 0m 41s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 0m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 36s Patch generated 73 ASF License warnings.
          47m 57s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:date2015-12-30
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779934/HDFS-9600-branch-2.6.patch
          JIRA Issue HDFS-9600
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f985a816dd21 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.6 / 0f9dd48
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          findbugs v1.3.9
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14011/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 71MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14011/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 10m 32s branch-2.6 passed -1 compile 0m 42s hadoop-hdfs in branch-2.6 failed with JDK v1.8.0_66. -1 compile 0m 44s hadoop-hdfs in branch-2.6 failed with JDK v1.7.0_91. +1 checkstyle 0m 21s branch-2.6 passed +1 mvnsite 0m 57s branch-2.6 passed +1 mvneclipse 0m 20s branch-2.6 passed -1 findbugs 2m 54s hadoop-hdfs-project/hadoop-hdfs in branch-2.6 has 272 extant Findbugs warnings. +1 javadoc 1m 11s branch-2.6 passed with JDK v1.8.0_66 +1 javadoc 1m 52s branch-2.6 passed with JDK v1.7.0_91 +1 mvninstall 0m 53s the patch passed -1 compile 0m 39s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 javac 0m 39s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 compile 0m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 javac 0m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 1s The patch has 2664 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 1m 10s The patch has 95 line(s) with tabs. +1 findbugs 2m 52s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 48s the patch passed with JDK v1.7.0_91 -1 unit 0m 41s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 0m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 36s Patch generated 73 ASF License warnings. 47m 57s Subsystem Report/Notes Docker Image:yetus/hadoop:date2015-12-30 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779934/HDFS-9600-branch-2.6.patch JIRA Issue HDFS-9600 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f985a816dd21 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.6 / 0f9dd48 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 compile https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt findbugs v1.3.9 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html compile https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14011/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14011/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 71MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14011/console This message was automatically generated.
          Hide
          yangzhe1991 Phil Yang added a comment -

          upload patches for branch-2, branch-2.7 and branch 2.6.

          In branch-2, the LocatedBlock returned by namenode.updateBlockForPipeline has no StorageIDs in it so I use the old StorageIDs in testcase.

          In branch-2.7, the parameter of isNeededReplication is Block, so we need get BlockInfoContiguous first for isComplete().

          In branch-2.6, we use BlockInfo just like 2.7's BlockInfoContiguous. And BlockManager.isNeededReplication is private that is not visible for testcase. So I change it to package visibility by removing private keyword.

          Show
          yangzhe1991 Phil Yang added a comment - upload patches for branch-2, branch-2.7 and branch 2.6. In branch-2, the LocatedBlock returned by namenode.updateBlockForPipeline has no StorageIDs in it so I use the old StorageIDs in testcase. In branch-2.7, the parameter of isNeededReplication is Block, so we need get BlockInfoContiguous first for isComplete(). In branch-2.6, we use BlockInfo just like 2.7's BlockInfoContiguous. And BlockManager.isNeededReplication is private that is not visible for testcase. So I change it to package visibility by removing private keyword.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Latest patch looks good. +1.

          I tried committing, but patch doesnt apply cleanly to branch-2 as well.

          Do I need make another patch for 2.6/2.7? Thanks.

          You need to create branch-2 and 2.7/2.6 patches as well.

          Show
          vinayrpet Vinayakumar B added a comment - Latest patch looks good. +1. I tried committing, but patch doesnt apply cleanly to branch-2 as well. Do I need make another patch for 2.6/2.7? Thanks. You need to create branch-2 and 2.7/2.6 patches as well.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 22s trunk passed
          +1 compile 1m 5s trunk passed with JDK v1.8.0_66
          +1 compile 0m 52s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 1m 6s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 2m 10s trunk passed
          +1 javadoc 1m 21s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 10s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 58s the patch passed
          +1 compile 1m 4s the patch passed with JDK v1.8.0_66
          +1 javac 1m 4s the patch passed
          +1 compile 0m 50s the patch passed with JDK v1.7.0_91
          +1 javac 0m 50s the patch passed
          -1 checkstyle 0m 21s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 164, now 164).
          +1 mvnsite 0m 59s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 12s the patch passed
          +1 javadoc 1m 6s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 46s the patch passed with JDK v1.7.0_91
          -1 unit 59m 28s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 55m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          145m 28s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
            hadoop.hdfs.server.namenode.TestNameNodeRecovery
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestDatanodeRegistration
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
            hadoop.hdfs.TestRenameWhileOpen
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
            hadoop.hdfs.TestFileCorruption



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779897/HDFS-9600-v4.patch
          JIRA Issue HDFS-9600
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b16fb8275d6a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ad997fa
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14008/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14008/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14008/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14008/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14008/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14008/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14008/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 22s trunk passed +1 compile 1m 5s trunk passed with JDK v1.8.0_66 +1 compile 0m 52s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 6s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 2m 10s trunk passed +1 javadoc 1m 21s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 10s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 58s the patch passed +1 compile 1m 4s the patch passed with JDK v1.8.0_66 +1 javac 1m 4s the patch passed +1 compile 0m 50s the patch passed with JDK v1.7.0_91 +1 javac 0m 50s the patch passed -1 checkstyle 0m 21s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 164, now 164). +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 12s the patch passed +1 javadoc 1m 6s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 46s the patch passed with JDK v1.7.0_91 -1 unit 59m 28s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 55m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 145m 28s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.namenode.TestNameNodeRecovery JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestDatanodeRegistration JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.TestRenameWhileOpen   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.TestFileCorruption Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779897/HDFS-9600-v4.patch JIRA Issue HDFS-9600 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b16fb8275d6a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ad997fa Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14008/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14008/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14008/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14008/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14008/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14008/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14008/console This message was automatically generated.
          Hide
          yangzhe1991 Phil Yang added a comment -

          upload v4 patch to remove checkstyle warnings and add cluster.waitActive().

          Do I need make another patch for 2.6/2.7? Thanks.

          Show
          yangzhe1991 Phil Yang added a comment - upload v4 patch to remove checkstyle warnings and add cluster.waitActive(). Do I need make another patch for 2.6/2.7? Thanks.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks Phil Yang for reporting this. The fix looks good to me.

          Nit for test: Should we call cluster.waitActive(); before operating the mini cluster?

          Show
          liuml07 Mingliang Liu added a comment - Thanks Phil Yang for reporting this. The fix looks good to me. Nit for test: Should we call cluster.waitActive(); before operating the mini cluster?
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks for the great find and patch Phil Yang.

          Meanwhile, could you fix checkstyle warnings too? both are line length > 80.

          +1 once addressed.

          Show
          vinayrpet Vinayakumar B added a comment - Thanks for the great find and patch Phil Yang . Meanwhile, could you fix checkstyle warnings too? both are line length > 80. +1 once addressed.
          Hide
          yangzhe1991 Phil Yang added a comment -

          I run tests that failed in the latest Jenkins report, they all pass on my own computer, I think they are unrelated with my patch

          Show
          yangzhe1991 Phil Yang added a comment - I run tests that failed in the latest Jenkins report, they all pass on my own computer, I think they are unrelated with my patch
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 24s trunk passed
          +1 compile 0m 39s trunk passed with JDK v1.8.0_66
          +1 compile 0m 40s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 53s trunk passed
          +1 javadoc 1m 7s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 49s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.8.0_66
          +1 javac 0m 39s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_91
          +1 javac 0m 38s the patch passed
          -1 checkstyle 0m 17s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 164, now 166).
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 1s the patch passed
          +1 javadoc 1m 4s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 47s the patch passed with JDK v1.7.0_91
          -1 unit 54m 11s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 54m 22s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          134m 42s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
            hadoop.hdfs.server.datanode.TestBlockScanner
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestWriteReadStripedFile
            hadoop.hdfs.server.namenode.TestNameNodeMXBean
            hadoop.hdfs.TestHDFSFileSystemContract



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779756/HDFS-9600-v3.patch
          JIRA Issue HDFS-9600
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 625db7d39dcd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d0a22ba
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14004/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14004/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14004/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14004/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14004/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14004/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14004/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 24s trunk passed +1 compile 0m 39s trunk passed with JDK v1.8.0_66 +1 compile 0m 40s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 1m 7s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 49s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 45s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_66 +1 javac 0m 39s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_91 +1 javac 0m 38s the patch passed -1 checkstyle 0m 17s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 164, now 166). +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 1s the patch passed +1 javadoc 1m 4s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_91 -1 unit 54m 11s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 54m 22s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 134m 42s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.server.datanode.TestBlockScanner JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.server.namenode.TestNameNodeMXBean   hadoop.hdfs.TestHDFSFileSystemContract Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779756/HDFS-9600-v3.patch JIRA Issue HDFS-9600 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 625db7d39dcd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d0a22ba Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14004/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14004/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14004/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14004/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14004/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14004/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14004/console This message was automatically generated.
          Hide
          yangzhe1991 Phil Yang added a comment -

          testDecommissionStatus failed, upload v3 patch to pass it.
          In DecommissionManager.isSufficient, it skips blocks which do not need replication and will remove the block from watching list. Now we can not skip by isNeededReplication.

          Show
          yangzhe1991 Phil Yang added a comment - testDecommissionStatus failed, upload v3 patch to pass it. In DecommissionManager.isSufficient, it skips blocks which do not need replication and will remove the block from watching list. Now we can not skip by isNeededReplication.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 37s trunk passed
          +1 compile 0m 53s trunk passed with JDK v1.8.0_66
          +1 compile 0m 44s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 55s trunk passed
          +1 javadoc 1m 8s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 54s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 52s the patch passed with JDK v1.8.0_66
          +1 javac 0m 52s the patch passed
          +1 compile 0m 43s the patch passed with JDK v1.7.0_91
          +1 javac 0m 43s the patch passed
          -1 checkstyle 0m 15s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 161, now 162).
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 0s the patch passed
          +1 javadoc 1m 3s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 42s the patch passed with JDK v1.7.0_91
          -1 unit 63m 41s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 61m 31s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          152m 41s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.namenode.TestDecommissioningStatus



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779638/HDFS-9600-v2.patch
          JIRA Issue HDFS-9600
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1d8bb4671749 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a0249da
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13999/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13999/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13999/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13999/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13999/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13999/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13999/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 37s trunk passed +1 compile 0m 53s trunk passed with JDK v1.8.0_66 +1 compile 0m 44s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 55s trunk passed +1 javadoc 1m 8s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 54s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 50s the patch passed +1 compile 0m 52s the patch passed with JDK v1.8.0_66 +1 javac 0m 52s the patch passed +1 compile 0m 43s the patch passed with JDK v1.7.0_91 +1 javac 0m 43s the patch passed -1 checkstyle 0m 15s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 161, now 162). +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 0s the patch passed +1 javadoc 1m 3s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 42s the patch passed with JDK v1.7.0_91 -1 unit 63m 41s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 61m 31s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 152m 41s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.TestDecommissioningStatus JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.namenode.TestDecommissioningStatus Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779638/HDFS-9600-v2.patch JIRA Issue HDFS-9600 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1d8bb4671749 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a0249da Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13999/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13999/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13999/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13999/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13999/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13999/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13999/console This message was automatically generated.
          Hide
          yangzhe1991 Phil Yang added a comment -

          Thanks!
          I find that branch-2.6 and branch-2.7 have a different interface on isNeededReplication, I'm not familiar with the process of contribution, should I submit another patch for 2.6/2.7 ?

          Show
          yangzhe1991 Phil Yang added a comment - Thanks! I find that branch-2.6 and branch-2.7 have a different interface on isNeededReplication, I'm not familiar with the process of contribution, should I submit another patch for 2.6/2.7 ?
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Indeed, two conditions are the same.

          +1 on the current patch. Will wait for the Jenkins report before committing the patch.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Indeed, two conditions are the same. +1 on the current patch. Will wait for the Jenkins report before committing the patch.
          Hide
          yangzhe1991 Phil Yang added a comment -

          Hi,
          Thanks for your reply. I'm not sure what is the difference between getUnderConstructionFeature() == null and isComplete(), I thought they equals each other so I use the simpler one. Is there any chance that isComplete()==false but we still check the replication or isComplete()==true but we should not check replication? Please correct me if I am wrong

          Show
          yangzhe1991 Phil Yang added a comment - Hi, Thanks for your reply. I'm not sure what is the difference between getUnderConstructionFeature() == null and isComplete(), I thought they equals each other so I use the simpler one. Is there any chance that isComplete()==false but we still check the replication or isComplete()==true but we should not check replication? Please correct me if I am wrong
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks for filing this bug and providing a patch. You are right that we should not check replication if the block is under construction. Good catch!

          For the patch, I think it should check if getUnderConstructionFeature() == null instead of isComplete(). Do you agree?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks for filing this bug and providing a patch. You are right that we should not check replication if the block is under construction. Good catch! For the patch, I think it should check if getUnderConstructionFeature() == null instead of isComplete(). Do you agree?
          Hide
          yangzhe1991 Phil Yang added a comment -

          Upload v2 patch which adds a testcase to reproduce this bug

          Show
          yangzhe1991 Phil Yang added a comment - Upload v2 patch which adds a testcase to reproduce this bug
          Hide
          yangzhe1991 Phil Yang added a comment -

          I find the reason of test case breaking and open a new issue to fix it on HDFS-9602

          Show
          yangzhe1991 Phil Yang added a comment - I find the reason of test case breaking and open a new issue to fix it on HDFS-9602
          Hide
          yangzhe1991 Phil Yang added a comment -

          The unit tests failed, but it seems that it is not caused by my patch, right?

          Show
          yangzhe1991 Phil Yang added a comment - The unit tests failed, but it seems that it is not caused by my patch, right?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 51s trunk passed
          +1 compile 0m 41s trunk passed with JDK v1.8.0_66
          +1 compile 0m 39s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 54s trunk passed
          +1 javadoc 1m 7s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 48s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 41s the patch passed with JDK v1.8.0_66
          +1 javac 0m 41s the patch passed
          +1 compile 0m 42s the patch passed with JDK v1.7.0_91
          +1 javac 0m 42s the patch passed
          -1 checkstyle 0m 15s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 161, now 162).
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 0s the patch passed
          +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 50s the patch passed with JDK v1.7.0_91
          -1 unit 73m 55s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 73m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 28s Patch does not generate ASF License warnings.
          174m 43s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.TestDecommissioningStatus



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779493/HDFS-9600-v1.patch
          JIRA Issue HDFS-9600
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 23b5d771b649 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6553784
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13995/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13995/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13995/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13995/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13995/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13995/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13995/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 51s trunk passed +1 compile 0m 41s trunk passed with JDK v1.8.0_66 +1 compile 0m 39s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 1m 7s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 48s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 51s the patch passed +1 compile 0m 41s the patch passed with JDK v1.8.0_66 +1 javac 0m 41s the patch passed +1 compile 0m 42s the patch passed with JDK v1.7.0_91 +1 javac 0m 42s the patch passed -1 checkstyle 0m 15s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 161, now 162). +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 0s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 50s the patch passed with JDK v1.7.0_91 -1 unit 73m 55s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 73m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 174m 43s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.TestDecommissioningStatus Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779493/HDFS-9600-v1.patch JIRA Issue HDFS-9600 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 23b5d771b649 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6553784 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13995/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13995/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13995/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13995/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13995/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13995/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13995/console This message was automatically generated.

            People

            • Assignee:
              yangzhe1991 Phil Yang
              Reporter:
              yangzhe1991 Phil Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development