Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9314

Improve BlockPlacementPolicyDefault's picking of excess replicas

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The test case used in HDFS-9313 identified NullPointerException as well as the limitation of excess replica picking. If the current replicas are on

      {SSD(rack r1), DISK(rack 2), DISK(rack 3), DISK(rack 3)}

      and the storage policy changes to HOT_STORAGE_POLICY_ID, BlockPlacementPolicyDefault's won't be able to delete SSD replica.

      1. HDFS-9314.001.patch
        3 kB
        Xiao Chen
      2. HDFS-9314.002.patch
        4 kB
        Xiao Chen
      3. HDFS-9314.003.patch
        7 kB
        Xiao Chen
      4. HDFS-9314.004.patch
        19 kB
        Xiao Chen
      5. HDFS-9314.005.patch
        21 kB
        Xiao Chen
      6. HDFS-9314.006.patch
        21 kB
        Xiao Chen
      7. HDFS-9314.branch26.patch
        15 kB
        Xiao Chen
      8. HDFS-9314.branch27.patch
        15 kB
        Xiao Chen

        Issue Links

          Activity

          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Ming Ma for creating this, and nice work on HDFS-9313!
          Looks like this is the same issue reported in HDFS-8770, as Walter Su pointed out in there.
          I attached an initial patch, following the idea ade posted in the other JIRA which makes sense to me. It passes locally for the 4 tests extended from BaseReplicationPolicyTest. Please review. Thank you.

          Show
          xiaochen Xiao Chen added a comment - Thanks Ming Ma for creating this, and nice work on HDFS-9313 ! Looks like this is the same issue reported in HDFS-8770 , as Walter Su pointed out in there. I attached an initial patch, following the idea ade posted in the other JIRA which makes sense to me. It passes locally for the 4 tests extended from BaseReplicationPolicyTest . Please review. Thank you.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the patch Xiao. Quick comment before I finish reviewing:

            protected DatanodeStorageInfo chooseReplicaToDelete(
                Collection<DatanodeStorageInfo> storages,
          

          Should be private?

          Show
          zhz Zhe Zhang added a comment - Thanks for the patch Xiao. Quick comment before I finish reviewing: protected DatanodeStorageInfo chooseReplicaToDelete( Collection<DatanodeStorageInfo> storages, Should be private?
          Hide
          zhz Zhe Zhang added a comment -

          The patch LGTM overall. Only ask is to add a short Javadoc for the new chooseReplicaToDelete method.

          I think the change fixes the issue for the default BPP. We should probably address it for BlockPlacementPolicyWithNodeGroup and BlockPlacementPolicyWithUpgradeDomain as follow-on tasks. In both subclasses pickupReplicaSet still returns one from two sets, and it's possible that the picked set doesn't have the appropriate type to delete.

          Show
          zhz Zhe Zhang added a comment - The patch LGTM overall. Only ask is to add a short Javadoc for the new chooseReplicaToDelete method. I think the change fixes the issue for the default BPP. We should probably address it for BlockPlacementPolicyWithNodeGroup and BlockPlacementPolicyWithUpgradeDomain as follow-on tasks. In both subclasses pickupReplicaSet still returns one from two sets, and it's possible that the picked set doesn't have the appropriate type to delete.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Zhe Zhang for the prompt review!
          Attached patch 002 to address your comments:

          • The new method should be private, since the method that's invoking it is VisibleForTesting and otherwise be private. Good catch.
          • Added Javadoc. I find the documents from BlockPlacementPolicy#chooseReplicasToDelete to be clearer to me, so derived from there.

          I think for the subclasses of BPP, since they only override pickupReplicaSet and do not override chooseReplicasToDelete, we can leave them as is. If in the future they need to override chooseReplicasToDelete and want to reuse some logic in BPP, they can extract and implement by then. Does this make sense?

          Show
          xiaochen Xiao Chen added a comment - Thanks Zhe Zhang for the prompt review! Attached patch 002 to address your comments: The new method should be private, since the method that's invoking it is VisibleForTesting and otherwise be private. Good catch. Added Javadoc. I find the documents from BlockPlacementPolicy#chooseReplicasToDelete to be clearer to me, so derived from there. I think for the subclasses of BPP, since they only override pickupReplicaSet and do not override chooseReplicasToDelete , we can leave them as is. If in the future they need to override chooseReplicasToDelete and want to reuse some logic in BPP, they can extract and implement by then. Does this make sense?
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Xiao! +1 pending Jenkins.

          My concern about other BPPs is that they potentially updates moreThanOne and exactlyOne. So directly feeding exactlyOne to the new chooseReplicaToDelete method bypasses the extra processing in pickupReplicaSet in subclasses. But I don't think this should block this JIRA anyway. Pinging Ming Ma for additional advice here.

          Show
          zhz Zhe Zhang added a comment - Thanks Xiao! +1 pending Jenkins. My concern about other BPPs is that they potentially updates moreThanOne and exactlyOne . So directly feeding exactlyOne to the new chooseReplicaToDelete method bypasses the extra processing in pickupReplicaSet in subclasses. But I don't think this should block this JIRA anyway. Pinging Ming Ma for additional advice here.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 6s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 35s trunk passed
          +1 compile 0m 38s trunk passed with JDK v1.8.0_60
          +1 compile 0m 36s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 0m 48s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 10s trunk passed
          +1 javadoc 1m 22s trunk passed with JDK v1.8.0_60
          +1 javadoc 2m 3s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 43s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.8.0_60
          +1 javac 0m 38s the patch passed
          +1 compile 0m 35s the patch passed with JDK v1.7.0_79
          +1 javac 0m 35s the patch passed
          +1 checkstyle 0m 17s the patch passed
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 19s the patch passed
          +1 javadoc 1m 15s the patch passed with JDK v1.8.0_60
          +1 javadoc 2m 6s the patch passed with JDK v1.7.0_79
          -1 unit 68m 49s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          -1 unit 58m 30s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 21s Patch generated 56 ASF License warnings.
          151m 23s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestEncryptionZones
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestRecoverStripedFile



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-16
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772537/HDFS-9314.001.patch
          JIRA Issue HDFS-9314
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cf3b37e68d5d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh
          git revision trunk / 02653ad
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13523/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13523/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13523/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13523/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13523/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13523/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 228MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13523/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 35s trunk passed +1 compile 0m 38s trunk passed with JDK v1.8.0_60 +1 compile 0m 36s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 48s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 10s trunk passed +1 javadoc 1m 22s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 3s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 43s the patch passed +1 compile 0m 38s the patch passed with JDK v1.8.0_60 +1 javac 0m 38s the patch passed +1 compile 0m 35s the patch passed with JDK v1.7.0_79 +1 javac 0m 35s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 19s the patch passed +1 javadoc 1m 15s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 6s the patch passed with JDK v1.7.0_79 -1 unit 68m 49s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 58m 30s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 21s Patch generated 56 ASF License warnings. 151m 23s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestEncryptionZones JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestRecoverStripedFile Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-16 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772537/HDFS-9314.001.patch JIRA Issue HDFS-9314 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cf3b37e68d5d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh git revision trunk / 02653ad findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13523/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13523/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13523/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13523/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13523/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13523/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 228MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13523/console This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          The test failures all seems unrelated.

          Show
          xiaochen Xiao Chen added a comment - The test failures all seems unrelated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 5s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 11s trunk passed
          +1 compile 0m 34s trunk passed with JDK v1.8.0_60
          +1 compile 0m 34s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 44s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 1m 8s trunk passed with JDK v1.8.0_60
          +1 javadoc 1m 51s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 41s the patch passed
          +1 compile 0m 35s the patch passed with JDK v1.8.0_60
          +1 javac 0m 35s the patch passed
          +1 compile 0m 34s the patch passed with JDK v1.7.0_79
          +1 javac 0m 34s the patch passed
          +1 checkstyle 0m 17s the patch passed
          +1 mvnsite 0m 43s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 9s the patch passed
          +1 javadoc 1m 11s the patch passed with JDK v1.8.0_60
          +1 javadoc 1m 54s the patch passed with JDK v1.7.0_79
          -1 unit 61m 10s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          -1 unit 61m 52s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 22s Patch generated 58 ASF License warnings.
          145m 6s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.TestWriteReadStripedFile
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
            hadoop.hdfs.TestAclsEndToEnd
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.datanode.TestBPOfferService
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-16
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772606/HDFS-9314.002.patch
          JIRA Issue HDFS-9314
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e28c9929d24a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh
          git revision trunk / 02653ad
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13524/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13524/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13524/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13524/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13524/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13524/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 225MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13524/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 5s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 11s trunk passed +1 compile 0m 34s trunk passed with JDK v1.8.0_60 +1 compile 0m 34s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 44s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 1m 8s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 51s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 41s the patch passed +1 compile 0m 35s the patch passed with JDK v1.8.0_60 +1 javac 0m 35s the patch passed +1 compile 0m 34s the patch passed with JDK v1.7.0_79 +1 javac 0m 34s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 43s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 1m 11s the patch passed with JDK v1.8.0_60 +1 javadoc 1m 54s the patch passed with JDK v1.7.0_79 -1 unit 61m 10s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 61m 52s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 22s Patch generated 58 ASF License warnings. 145m 6s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.TestAclsEndToEnd JDK v1.7.0_79 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.datanode.TestBPOfferService   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-16 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772606/HDFS-9314.002.patch JIRA Issue HDFS-9314 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e28c9929d24a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh git revision trunk / 02653ad findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13524/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13524/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13524/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13524/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13524/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13524/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 225MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13524/console This message was automatically generated.
          Hide
          walter.k.su Walter Su added a comment -

          Shall we consider not to violate the placement policy when we decide to decrease rack counts?
          In HDFS-9313 I agree to choose the SSD, since the remaining 3 DISKs are already on enough racks.

          Consider a different case,

          rack_0 rack_1
          replica_0(DISK) replica_3(SSD)
          replica_1(DISK)
          replica_2(DISK)

          If we remove replica_3, the left are not on enough rack. Is it removing replica_3 the right decision any more? Shall we make sure not to violate the placement policy, before decide to decrease rack counts? We can schedule a replication task first, then come back to delete the SSD.
          The old logic will choose from exactlyOne collection as well, but only if moreThanOne is empty. So the old logic doesn't violate the placement policy.

          Show
          walter.k.su Walter Su added a comment - Shall we consider not to violate the placement policy when we decide to decrease rack counts? In HDFS-9313 I agree to choose the SSD, since the remaining 3 DISKs are already on enough racks. Consider a different case, rack_0 rack_1 replica_0(DISK) replica_3(SSD) replica_1(DISK) replica_2(DISK) If we remove replica_3, the left are not on enough rack. Is it removing replica_3 the right decision any more? Shall we make sure not to violate the placement policy, before decide to decrease rack counts? We can schedule a replication task first, then come back to delete the SSD. The old logic will choose from exactlyOne collection as well, but only if moreThanOne is empty. So the old logic doesn't violate the placement policy.
          Hide
          mingma Ming Ma added a comment -

          Thanks @Xiao Chen for working on this. I agree with Zhe Zhang and Walter Su, this could end up violating block placement policy, as the fallback chooseReplicaToDelete call uses the remaining replicas which aren't honored by the block placement policy.

          • Another way to address this, maybe we can modify the semantics of BlockPlacementPolicyDefault#pickupReplicaSet to include "all replicas that deletion of the replica won't violate the policy". So using the example listed in the description, it will return all 4 replicas instead given deletion of any one of them is ok given the number of racks >=2 afterwards. This does change the existing semantics of "making sure the number of racks remains unchanged after deletion." But it should be ok given BlockPlacementPolicyDefault only guarantees "the number of racks >= 2" policy. With this approach, there is no need for the fallback call.
          • We still have the case like what Water mentioned above where it just can't pick any replica to satisfy the block placement policy. IMHO, we can follow up in a separate jira as that might require more changes to schedule replication etc. This jira can just cover the case single pass can solve and make sure it doesn't suggest replica that will violate block placement policy.
          Show
          mingma Ming Ma added a comment - Thanks @Xiao Chen for working on this. I agree with Zhe Zhang and Walter Su , this could end up violating block placement policy, as the fallback chooseReplicaToDelete call uses the remaining replicas which aren't honored by the block placement policy. Another way to address this, maybe we can modify the semantics of BlockPlacementPolicyDefault#pickupReplicaSet to include "all replicas that deletion of the replica won't violate the policy". So using the example listed in the description, it will return all 4 replicas instead given deletion of any one of them is ok given the number of racks >=2 afterwards. This does change the existing semantics of "making sure the number of racks remains unchanged after deletion." But it should be ok given BlockPlacementPolicyDefault only guarantees "the number of racks >= 2" policy. With this approach, there is no need for the fallback call. We still have the case like what Water mentioned above where it just can't pick any replica to satisfy the block placement policy. IMHO, we can follow up in a separate jira as that might require more changes to schedule replication etc. This jira can just cover the case single pass can solve and make sure it doesn't suggest replica that will violate block placement policy.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Ming Ma, Walter Su and Zhe Zhang for the valuable inputs! My initial patch failed to consider the block placement policy of keeping replicas on different racks.
          To handle that, I agree that BlockPlacementPolicyDefault#pickupReplicaSet would be a better place to go, for the following reasons:

          • Doing it here would keep the delete logic unchanged, hence not impacting subclasses.
          • This makes it easier to separate logic in the future if we want to schedule for rebalance
          • The nature of the problem is that we didn't choose from the correct racks to delete, which is what this method is doing.

          Attached patch 3 handles the issue in pickupReplicaSet.

          • In rack name comparison, I used hashCode to be consistent with the fact we used HashMap in splitNodesWithRack to do the same thing. (It would be ideal if we can pass in rackMap object, but that needs API change.)
          • As Ming mentioned, this implementation changes the existing semantics of "making sure the number of racks remains unchanged after deletion". So I have to update unit test in testChooseReplicasToDelete to pass. The sequence of the 2 deletes changed, but the final result is the same.
          • I also added 2 new tests in testChooseReplicasToDelete with explanation, covering some sample scenarios.

          I have created HDFS-9440 to handle the case where we cannot pick any replica without violating the block placement policy.
          Please review and provide your feedback. Thanks!

          Show
          xiaochen Xiao Chen added a comment - Thanks Ming Ma , Walter Su and Zhe Zhang for the valuable inputs! My initial patch failed to consider the block placement policy of keeping replicas on different racks. To handle that, I agree that BlockPlacementPolicyDefault#pickupReplicaSet would be a better place to go, for the following reasons: Doing it here would keep the delete logic unchanged, hence not impacting subclasses. This makes it easier to separate logic in the future if we want to schedule for rebalance The nature of the problem is that we didn't choose from the correct racks to delete, which is what this method is doing. Attached patch 3 handles the issue in pickupReplicaSet . In rack name comparison, I used hashCode to be consistent with the fact we used HashMap in splitNodesWithRack to do the same thing. (It would be ideal if we can pass in rackMap object, but that needs API change.) As Ming mentioned, this implementation changes the existing semantics of "making sure the number of racks remains unchanged after deletion". So I have to update unit test in testChooseReplicasToDelete to pass. The sequence of the 2 deletes changed, but the final result is the same. I also added 2 new tests in testChooseReplicasToDelete with explanation, covering some sample scenarios. I have created HDFS-9440 to handle the case where we cannot pick any replica without violating the block placement policy. Please review and provide your feedback. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 43s trunk passed
          +1 compile 0m 57s trunk passed with JDK v1.8.0_66
          +1 compile 0m 50s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 2m 13s trunk passed
          +1 javadoc 1m 25s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 14s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 58s the patch passed
          +1 compile 0m 58s the patch passed with JDK v1.8.0_66
          +1 javac 0m 58s the patch passed
          +1 compile 0m 49s the patch passed with JDK v1.7.0_79
          +1 javac 0m 49s the patch passed
          +1 checkstyle 0m 18s the patch passed
          +1 mvnsite 1m 2s the patch passed
          +1 mvneclipse 0m 16s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 29s the patch passed
          +1 javadoc 1m 28s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 14s the patch passed with JDK v1.7.0_79
          -1 unit 76m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 72m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 21s Patch generated 56 ASF License warnings.
          181m 4s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.cli.TestHDFSCLI
            hadoop.hdfs.TestLeaseRecovery2
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.qjournal.client.TestQuorumJournalManager
            hadoop.hdfs.server.namenode.TestCacheDirectives
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDNFencing
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:date2015-11-19
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773141/HDFS-9314.003.patch
          JIRA Issue HDFS-9314
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c90f87162805 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh
          git revision trunk / 23a130a
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13556/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13556/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13556/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13556/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13556/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13556/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 78MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13556/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 43s trunk passed +1 compile 0m 57s trunk passed with JDK v1.8.0_66 +1 compile 0m 50s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 19s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 13s trunk passed +1 javadoc 1m 25s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 14s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 58s the patch passed +1 compile 0m 58s the patch passed with JDK v1.8.0_66 +1 javac 0m 58s the patch passed +1 compile 0m 49s the patch passed with JDK v1.7.0_79 +1 javac 0m 49s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 1m 2s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 29s the patch passed +1 javadoc 1m 28s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 14s the patch passed with JDK v1.7.0_79 -1 unit 76m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 72m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 21s Patch generated 56 ASF License warnings. 181m 4s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.cli.TestHDFSCLI   hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.qjournal.client.TestQuorumJournalManager   hadoop.hdfs.server.namenode.TestCacheDirectives JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles Subsystem Report/Notes Docker Image:yetus/hadoop:date2015-11-19 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773141/HDFS-9314.003.patch JIRA Issue HDFS-9314 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c90f87162805 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh git revision trunk / 23a130a findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13556/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13556/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13556/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13556/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13556/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13556/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 78MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13556/console This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Thanks Xiao Chen. This should work.

          • The rack-to-hosts mapping rackMap has already been computed. Do you think it is clearer if we modify pickupReplicaSet to take the rack-to-hosts mapping rackMap as input instead? In that way, it can say something like: a) if number of existing replicas is 2, both nodes are candidates. b) if number of racks >=3, all nodes are candidates. c) if the number of racks <=2, all nodes that share racks with at least another node are candidates.
          • As far as I can tell, this shouldn't impact BlockPlacementPolicyWithUpgradeDomain. It will be useful if you can add a new test case to TestReplicationPolicyWithUpgradeDomain to make sure it works. Also the description for BlockPlacementPolicyWithUpgradeDomain's pickupReplicaSet will need some update as BlockPlacementPolicyDefault's pickupReplicaSet no longer only return set of share-rack nodes anymore; instead it will return a super set of share-rack nodes.
          Show
          mingma Ming Ma added a comment - Thanks Xiao Chen . This should work. The rack-to-hosts mapping rackMap has already been computed. Do you think it is clearer if we modify pickupReplicaSet to take the rack-to-hosts mapping rackMap as input instead? In that way, it can say something like: a) if number of existing replicas is 2, both nodes are candidates. b) if number of racks >=3, all nodes are candidates. c) if the number of racks <=2, all nodes that share racks with at least another node are candidates. As far as I can tell, this shouldn't impact BlockPlacementPolicyWithUpgradeDomain . It will be useful if you can add a new test case to TestReplicationPolicyWithUpgradeDomain to make sure it works. Also the description for BlockPlacementPolicyWithUpgradeDomain 's pickupReplicaSet will need some update as BlockPlacementPolicyDefault 's pickupReplicaSet no longer only return set of share-rack nodes anymore; instead it will return a super set of share-rack nodes.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Ming Ma for the review, I'll be working on the next rev.
          1 quick clarification: I thought about passing in rackMap, but that needs to add the extra parameter to both chooseReplicaToDelete and pickupReplicaSet.

          • I guess @VisibleForTesting public chooseReplicaToDelete should be fine.
          • But changing the protected pickupReplicaSet would require modifying all subclasses of BlockPlacementPolicyDefault, which is marked as @InterfaceAudience.Private so no external subclasses are expected. So we'll just update all usages in current code and not worry about compatibility.

          Are we on the same page here?

          Show
          xiaochen Xiao Chen added a comment - Thanks Ming Ma for the review, I'll be working on the next rev. 1 quick clarification: I thought about passing in rackMap , but that needs to add the extra parameter to both chooseReplicaToDelete and pickupReplicaSet . I guess @VisibleForTesting public chooseReplicaToDelete should be fine. But changing the protected pickupReplicaSet would require modifying all subclasses of BlockPlacementPolicyDefault , which is marked as @InterfaceAudience.Private so no external subclasses are expected . So we'll just update all usages in current code and not worry about compatibility. Are we on the same page here?
          Hide
          mingma Ming Ma added a comment -

          Xiao Chen, AFAIK, we can assume there is no much compatibility requirement for BlockPlacementPolicy. Recently we have made several changes.

          Show
          mingma Ming Ma added a comment - Xiao Chen , AFAIK, we can assume there is no much compatibility requirement for BlockPlacementPolicy. Recently we have made several changes.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks again Ming Ma! Attached patch 4 with the aforementioned changes:

          • Pass in rackMap to pickupReplicaSet, add param to methods along the way.
          • Update comments in BlockPlacementPolicyWithUpgradeDomain.
          • Add test case to TestReplicationPolicyWithUpgradeDomain, verified it failed before the fix and passed after.
          Show
          xiaochen Xiao Chen added a comment - Thanks again Ming Ma ! Attached patch 4 with the aforementioned changes: Pass in rackMap to pickupReplicaSet , add param to methods along the way. Update comments in BlockPlacementPolicyWithUpgradeDomain . Add test case to TestReplicationPolicyWithUpgradeDomain , verified it failed before the fix and passed after.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 8m 20s trunk passed
          +1 compile 0m 48s trunk passed with JDK v1.8.0_66
          +1 compile 0m 45s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 2m 6s trunk passed
          +1 javadoc 1m 15s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 58s trunk passed with JDK v1.7.0_85
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 46s the patch passed with JDK v1.8.0_66
          +1 javac 0m 46s the patch passed
          +1 compile 0m 46s the patch passed with JDK v1.7.0_85
          +1 javac 0m 46s the patch passed
          -1 checkstyle 0m 17s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 49, now 50).
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 16s the patch passed
          +1 javadoc 1m 15s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 57s the patch passed with JDK v1.7.0_85
          -1 unit 57m 58s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 54m 10s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
          -1 asflicense 0m 21s Patch generated 58 ASF License warnings.
          141m 27s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
            hadoop.hdfs.server.namenode.TestBackupNode
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestDataTransferKeepalive
          JDK v1.7.0_85 Failed junit tests hadoop.fs.TestUnbuffer
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:date2015-11-20
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773585/HDFS-9314.004.patch
          JIRA Issue HDFS-9314
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8628c337d923 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh
          git revision trunk / 4539131
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13584/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13584/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13584/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13584/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13584/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13584/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13584/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 78MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13584/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 8m 20s trunk passed +1 compile 0m 48s trunk passed with JDK v1.8.0_66 +1 compile 0m 45s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 6s trunk passed +1 javadoc 1m 15s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 58s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 51s the patch passed +1 compile 0m 46s the patch passed with JDK v1.8.0_66 +1 javac 0m 46s the patch passed +1 compile 0m 46s the patch passed with JDK v1.7.0_85 +1 javac 0m 46s the patch passed -1 checkstyle 0m 17s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 49, now 50). +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 16s the patch passed +1 javadoc 1m 15s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 57s the patch passed with JDK v1.7.0_85 -1 unit 57m 58s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 54m 10s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 21s Patch generated 58 ASF License warnings. 141m 27s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.server.namenode.TestBackupNode   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestDataTransferKeepalive JDK v1.7.0_85 Failed junit tests hadoop.fs.TestUnbuffer   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Subsystem Report/Notes Docker Image:yetus/hadoop:date2015-11-20 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773585/HDFS-9314.004.patch JIRA Issue HDFS-9314 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8628c337d923 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh git revision trunk / 4539131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13584/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13584/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13584/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13584/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13584/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13584/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13584/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 78MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13584/console This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Thanks.

          • If there are two replicas node1(rack 1) and node2(rack 2), pickupReplicaSet should still return both nodes. Maybe it is useful to have a new test for this special case.
          • Nit: Comment {{storages ( {storages[3]}, {storages[7], storages[8]}) are on different racks (r1, r2, r3)}} should be {{storages ({storages[3]}

            ,

            {storages[7], storages[8]}

            ) are on different racks (r2, r3, r3)}}.

          Show
          mingma Ming Ma added a comment - Thanks. If there are two replicas node1(rack 1) and node2(rack 2), pickupReplicaSet should still return both nodes. Maybe it is useful to have a new test for this special case. Nit: Comment {{storages ( {storages[3]}, {storages[7], storages[8]}) are on different racks (r1, r2, r3)}} should be {{storages ({storages[3]} , {storages[7], storages[8]} ) are on different racks (r2, r3, r3)}}.
          Hide
          walter.k.su Walter Su added a comment -

          Except the 2 replicas scenario Ming Ma mentioned, the changes to default policy looks good to me.
          1. Do you mind copy the old pickupReplicaSet logic to BlockPlacementPolicyRackFaultTolerant? It still expects not to reduce rack counts, as possible.
          2. BlockPlacementPolicyWithNodeGroup prefers to pick nodes on the same node-group. If no such nodes, the logic has no difference with the default policy, before we change the default policy. So BlockPlacementPolicyWithNodeGroup needs to be fixed too.

          Show
          walter.k.su Walter Su added a comment - Except the 2 replicas scenario Ming Ma mentioned, the changes to default policy looks good to me. 1. Do you mind copy the old pickupReplicaSet logic to BlockPlacementPolicyRackFaultTolerant ? It still expects not to reduce rack counts, as possible. 2. BlockPlacementPolicyWithNodeGroup prefers to pick nodes on the same node-group. If no such nodes, the logic has no difference with the default policy, before we change the default policy. So BlockPlacementPolicyWithNodeGroup needs to be fixed too.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the review Ming and Walter.
          Patch 5 addressed both points in Ming's comments and Walter's comment #1.
          Walter Su, for BlockPlacementPolicyWithNodeGroup, I think the existing code is ok. IIUC the code is splitting nodes depending on group names inside pickupReplicaSet. After the split it tries to return moreThanOne if not empty, otherwise exactlyOne. This was done by directly copying the old implementation rather than calling super.pickupReplicaSet. So we don't need the default pickupReplicaSet in either case now, correct?

          Show
          xiaochen Xiao Chen added a comment - Thanks for the review Ming and Walter. Patch 5 addressed both points in Ming's comments and Walter's comment #1. Walter Su , for BlockPlacementPolicyWithNodeGroup , I think the existing code is ok. IIUC the code is splitting nodes depending on group names inside pickupReplicaSet . After the split it tries to return moreThanOne if not empty, otherwise exactlyOne . This was done by directly copying the old implementation rather than calling super.pickupReplicaSet . So we don't need the default pickupReplicaSet in either case now, correct?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 9m 50s trunk passed
          +1 compile 1m 5s trunk passed with JDK v1.8.0_66
          +1 compile 0m 56s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 1m 9s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 2m 32s trunk passed
          +1 javadoc 1m 31s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 31s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 8s the patch passed
          +1 compile 1m 7s the patch passed with JDK v1.8.0_66
          +1 javac 1m 7s the patch passed
          +1 compile 0m 57s the patch passed with JDK v1.7.0_85
          +1 javac 0m 57s the patch passed
          +1 checkstyle 0m 22s the patch passed
          +1 mvnsite 1m 12s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 52s the patch passed
          +1 javadoc 1m 41s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 30s the patch passed with JDK v1.7.0_85
          -1 unit 110m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 98m 5s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
          -1 asflicense 0m 29s Patch generated 56 ASF License warnings.
          244m 53s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.ha.TestDNFencing
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.mover.TestStorageMover
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.namenode.TestNameNodeAcl
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
            hadoop.hdfs.server.namenode.TestAddStripedBlocks
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.namenode.TestFileContextXAttr
            hadoop.hdfs.TestFileCreationClient
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773742/HDFS-9314.005.patch
          JIRA Issue HDFS-9314
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e36ad23f16c1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a6f20d8
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13597/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13597/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13597/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13597/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13597/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13597/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13597/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 9m 50s trunk passed +1 compile 1m 5s trunk passed with JDK v1.8.0_66 +1 compile 0m 56s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 20s trunk passed +1 mvnsite 1m 9s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 32s trunk passed +1 javadoc 1m 31s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 31s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 8s the patch passed +1 compile 1m 7s the patch passed with JDK v1.8.0_66 +1 javac 1m 7s the patch passed +1 compile 0m 57s the patch passed with JDK v1.7.0_85 +1 javac 0m 57s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 1m 12s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 52s the patch passed +1 javadoc 1m 41s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 30s the patch passed with JDK v1.7.0_85 -1 unit 110m 1s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 98m 5s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 29s Patch generated 56 ASF License warnings. 244m 53s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.mover.TestStorageMover   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.TestNameNodeAcl   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.server.namenode.TestAddStripedBlocks   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.namenode.TestFileContextXAttr   hadoop.hdfs.TestFileCreationClient   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773742/HDFS-9314.005.patch JIRA Issue HDFS-9314 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e36ad23f16c1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a6f20d8 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13597/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13597/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13597/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13597/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13597/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13597/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13597/console This message was automatically generated.
          Hide
          walter.k.su Walter Su added a comment -

          Suppose first, second are not empty.

          Before changes to default policy:
          1. choose from first.

          After changes to default policy:
          1. choose from first.
          2. If step.1 returns null, fallback to choose from second.

          Suppose first, second are not empty, and suppose in first there are nodes on same node-group
          Before changes to node-group policy:
          1. choose from nodes belonging to same node-group from first.

          After changes to node-group policy (What I thought I should be):
          1. choose from nodes belonging to same node-group from first.
          2. If step.1 returns null, fallback to all first.
          3. If step.2 returns null, fallback to choose from second.
          (The step.2~3 is the same as default policy)

          Satisfying the placement policy is the prerequisite for the fallback strategy.

          What I meant is, we support fallback in default policy, why not node-group policy?
          Besides, the 004 patch simply returns all, there is no preference to first or second. We can't say it's a fallback strategy.

          Show
          walter.k.su Walter Su added a comment - Suppose first , second are not empty. Before changes to default policy: 1. choose from first . After changes to default policy: 1. choose from first . 2. If step.1 returns null, fallback to choose from second . Suppose first, second are not empty, and suppose in first there are nodes on same node-group Before changes to node-group policy: 1. choose from nodes belonging to same node-group from first . After changes to node-group policy (What I thought I should be): 1. choose from nodes belonging to same node-group from first . 2. If step.1 returns null, fallback to all first . 3. If step.2 returns null, fallback to choose from second . (The step.2~3 is the same as default policy) Satisfying the placement policy is the prerequisite for the fallback strategy. What I meant is, we support fallback in default policy, why not node-group policy? Besides, the 004 patch simply returns all , there is no preference to first or second . We can't say it's a fallback strategy.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Walter.
          From patch 3, the implementation is no longer as a fallback strategy, but as a guarantee-the-number-of-remaining-racks-don't-go-<2 strategy. See comments above between Ming and me for details about this decision.
          So after the changes to default policy:

            /* If only 1 rack, pick all. If 2 racks, pick all that have more than
             * 1 replicas on the same rack; if no such replicas, pick all.
             * If 3 or more racks, pick all.
             */
          

          Above said, currently the node-group policy favors first with node-group specific logic as long as first is not empty. Then when choosing from moreThanOne and exactlyOne, we could apply default logic here, but instead of passing in rackMap we pass in nodeGroupMap. I'm not sure from requirement perspective if this is acceptable, but it would be more consistent logically. Makes sense? Also asking Ming Ma for advice.

          Attached patch 7 implements this idea. FYI - the only difference between patch 6 and 7 is the following:

          hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
          @@ -367,7 +367,7 @@ private int addDependentNodesToExcludedNodes(DatanodeDescriptor chosenNode,
          -    return moreThanOne.isEmpty()? exactlyOne : moreThanOne;
          +    return super.pickupReplicaSet(moreThanOne, exactlyOne, nodeGroupMap);
             }
          
          Show
          xiaochen Xiao Chen added a comment - Thanks Walter. From patch 3, the implementation is no longer as a fallback strategy, but as a guarantee-the-number-of-remaining-racks-don't-go-<2 strategy. See comments above between Ming and me for details about this decision. So after the changes to default policy: /* If only 1 rack, pick all. If 2 racks, pick all that have more than * 1 replicas on the same rack; if no such replicas, pick all. * If 3 or more racks, pick all. */ Above said, currently the node-group policy favors first with node-group specific logic as long as first is not empty. Then when choosing from moreThanOne and exactlyOne , we could apply default logic here, but instead of passing in rackMap we pass in nodeGroupMap . I'm not sure from requirement perspective if this is acceptable, but it would be more consistent logically. Makes sense? Also asking Ming Ma for advice. Attached patch 7 implements this idea. FYI - the only difference between patch 6 and 7 is the following: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java @@ -367,7 +367,7 @@ private int addDependentNodesToExcludedNodes(DatanodeDescriptor chosenNode, - return moreThanOne.isEmpty()? exactlyOne : moreThanOne; + return super .pickupReplicaSet(moreThanOne, exactlyOne, nodeGroupMap); }
          Hide
          xiaochen Xiao Chen added a comment -

          Sorry typo above, attached is patch 6, not 7.

          Show
          xiaochen Xiao Chen added a comment - Sorry typo above, attached is patch 6, not 7.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 10m 27s trunk passed
          +1 compile 1m 11s trunk passed with JDK v1.8.0_66
          +1 compile 1m 0s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 2m 29s trunk passed
          +1 javadoc 1m 33s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 23s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 5s the patch passed
          +1 compile 1m 2s the patch passed with JDK v1.8.0_66
          +1 javac 1m 3s the patch passed
          +1 compile 0m 57s the patch passed with JDK v1.7.0_85
          +1 javac 0m 57s the patch passed
          +1 checkstyle 0m 21s the patch passed
          +1 mvnsite 1m 14s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 50s the patch passed
          +1 javadoc 1m 40s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 50s the patch passed with JDK v1.7.0_85
          -1 unit 65m 6s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 82m 26s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
          -1 asflicense 0m 21s Patch generated 56 ASF License warnings.
          184m 55s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.namenode.TestFSImage
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.namenode.ha.TestHASafeMode
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.security.TestPermission
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestPread
            org.apache.hadoop.hdfs.TestSafeMode
            org.apache.hadoop.hdfs.TestFileAppend4
            org.apache.hadoop.hdfs.TestDFSUpgrade
          JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.datanode.TestRefreshNamenodes
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
            hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.security.TestPermission



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773774/HDFS-9314.006.patch
          JIRA Issue HDFS-9314
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f5fdc0cc9035 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 611aa77
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13611/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13611/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13611/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13611/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13611/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13611/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13611/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 10m 27s trunk passed +1 compile 1m 11s trunk passed with JDK v1.8.0_66 +1 compile 1m 0s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 20s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 29s trunk passed +1 javadoc 1m 33s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 23s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 5s the patch passed +1 compile 1m 2s the patch passed with JDK v1.8.0_66 +1 javac 1m 3s the patch passed +1 compile 0m 57s the patch passed with JDK v1.7.0_85 +1 javac 0m 57s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 1m 14s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 50s the patch passed +1 javadoc 1m 40s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 50s the patch passed with JDK v1.7.0_85 -1 unit 65m 6s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 82m 26s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 21s Patch generated 56 ASF License warnings. 184m 55s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.ha.TestHASafeMode   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.security.TestPermission   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestPread   org.apache.hadoop.hdfs.TestSafeMode   org.apache.hadoop.hdfs.TestFileAppend4   org.apache.hadoop.hdfs.TestDFSUpgrade JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.datanode.TestRefreshNamenodes   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.security.TestPermission Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773774/HDFS-9314.006.patch JIRA Issue HDFS-9314 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f5fdc0cc9035 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 611aa77 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13611/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13611/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13611/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13611/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13611/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13611/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13611/console This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          The approach of the jira is to relax the policy regarding which replica can be deleted so that it has more replicas to choose from to satisfy other constraints such as storage type. Implementation wise, we modify the semantics of pickupReplicaSet from "return the best replicas" to "return all replicas as long as it doesn't violate the block placement policy".

          For the question regarding BlockPlacementPolicyWithNodeGroup's pickupReplicaSet, if we apply the new semantics of pickupReplicaSet, given BlockPlacementPolicyWithNodeGroup reuses BlockPlacementPolicyDefault's verifyBlockPlacement, does BlockPlacementPolicyWithNodeGroup still need to have its own pickupReplicaSet?

          Show
          mingma Ming Ma added a comment - The approach of the jira is to relax the policy regarding which replica can be deleted so that it has more replicas to choose from to satisfy other constraints such as storage type. Implementation wise, we modify the semantics of pickupReplicaSet from "return the best replicas" to "return all replicas as long as it doesn't violate the block placement policy". For the question regarding BlockPlacementPolicyWithNodeGroup 's pickupReplicaSet , if we apply the new semantics of pickupReplicaSet , given BlockPlacementPolicyWithNodeGroup reuses BlockPlacementPolicyDefault 's verifyBlockPlacement , does BlockPlacementPolicyWithNodeGroup still need to have its own pickupReplicaSet ?
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Ming Ma, good point on verifyBlockPlacement!
          It's true that BlockPlacementPolicyWithNodeGroup does not override verifyBlockPlacement, but BlockPlacementPolicyWithNodeGroup#pickupReplicaSet favors node group name when choosing replicas for delete. I'm not sure replacing it with default is better, or adding BlockPlacementPolicyWithNodeGroup#verifyBlockPlacement. Do you think it's safe to remove it? Perhaps that's a separate discussion... Also adding original author Junping Du for more insights.

          Show
          xiaochen Xiao Chen added a comment - Thanks Ming Ma , good point on verifyBlockPlacement ! It's true that BlockPlacementPolicyWithNodeGroup does not override verifyBlockPlacement , but BlockPlacementPolicyWithNodeGroup#pickupReplicaSet favors node group name when choosing replicas for delete. I'm not sure replacing it with default is better, or adding BlockPlacementPolicyWithNodeGroup#verifyBlockPlacement . Do you think it's safe to remove it? Perhaps that's a separate discussion... Also adding original author Junping Du for more insights.
          Hide
          djp Junping Du added a comment -

          Let's keep BlockPlacementPolicyWithNodeGroup#pickupReplicaSet there, otherwise it will mess up block placement after deleting extra replicas. Actually, I think it is better to make BlockPlacementPolicyWithNodeGroup override verifyBlockPlacement() to check status according to its own policy. I will file a separated JIRA to track this.

          Show
          djp Junping Du added a comment - Let's keep BlockPlacementPolicyWithNodeGroup#pickupReplicaSet there, otherwise it will mess up block placement after deleting extra replicas. Actually, I think it is better to make BlockPlacementPolicyWithNodeGroup override verifyBlockPlacement() to check status according to its own policy. I will file a separated JIRA to track this.
          Hide
          djp Junping Du added a comment -

          Filed HDFS-9456.

          Show
          djp Junping Du added a comment - Filed HDFS-9456 .
          Hide
          mingma Ming Ma added a comment -

          +1 on patch 5 given Junping Du's comments. Xiao Chen, can you please confirm if the unit test failures weren't related?

          Show
          mingma Ming Ma added a comment - +1 on patch 5 given Junping Du 's comments. Xiao Chen , can you please confirm if the unit test failures weren't related?
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Junping Du and Ming Ma.
          The failed tests in patch 5 are not related. I have locally run the tests which appeared to be failing on both (intersect) JDK 7 & 8, and all of that passed. The environment seems to be having problems as there's a lot of timeouts in the failures. Also a couple of flaky tests such as HDFS-9300.

          Show
          xiaochen Xiao Chen added a comment - Thanks Junping Du and Ming Ma . The failed tests in patch 5 are not related. I have locally run the tests which appeared to be failing on both (intersect) JDK 7 & 8, and all of that passed. The environment seems to be having problems as there's a lot of timeouts in the failures. Also a couple of flaky tests such as HDFS-9300 .
          Hide
          mingma Ming Ma added a comment -

          Committed patch 5 to trunk and branch-2. Thanks Xiao Chen for the contribution. Thanks Walter Su Zhe Zhang Junping Du for the review.

          Show
          mingma Ming Ma added a comment - Committed patch 5 to trunk and branch-2. Thanks Xiao Chen for the contribution. Thanks Walter Su Zhe Zhang Junping Du for the review.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8876 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8876/)
          HDFS-9314. Improve BlockPlacementPolicyDefault's picking of excess (mingma: rev 0e54b164a8d8acf09aca8712116bf7a554cb4846)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8876 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8876/ ) HDFS-9314 . Improve BlockPlacementPolicyDefault's picking of excess (mingma: rev 0e54b164a8d8acf09aca8712116bf7a554cb4846) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks very much Ming Ma for the reviews and commit!
          Thanks Zhe, Walter and Junping for the reviews / suggestions.

          Show
          xiaochen Xiao Chen added a comment - Thanks very much Ming Ma for the reviews and commit! Thanks Zhe, Walter and Junping for the reviews / suggestions.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #725 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/725/)
          HDFS-9314. Improve BlockPlacementPolicyDefault's picking of excess (mingma: rev 0e54b164a8d8acf09aca8712116bf7a554cb4846)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #725 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/725/ ) HDFS-9314 . Improve BlockPlacementPolicyDefault's picking of excess (mingma: rev 0e54b164a8d8acf09aca8712116bf7a554cb4846) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2656 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2656/)
          HDFS-9314. Improve BlockPlacementPolicyDefault's picking of excess (mingma: rev 0e54b164a8d8acf09aca8712116bf7a554cb4846)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2656 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2656/ ) HDFS-9314 . Improve BlockPlacementPolicyDefault's picking of excess (mingma: rev 0e54b164a8d8acf09aca8712116bf7a554cb4846) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #715 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/715/)
          HDFS-9314. Improve BlockPlacementPolicyDefault's picking of excess (mingma: rev 0e54b164a8d8acf09aca8712116bf7a554cb4846)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #715 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/715/ ) HDFS-9314 . Improve BlockPlacementPolicyDefault's picking of excess (mingma: rev 0e54b164a8d8acf09aca8712116bf7a554cb4846) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1448 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1448/)
          HDFS-9314. Improve BlockPlacementPolicyDefault's picking of excess (mingma: rev 0e54b164a8d8acf09aca8712116bf7a554cb4846)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1448 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1448/ ) HDFS-9314 . Improve BlockPlacementPolicyDefault's picking of excess (mingma: rev 0e54b164a8d8acf09aca8712116bf7a554cb4846) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #638 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/638/)
          HDFS-9314. Improve BlockPlacementPolicyDefault's picking of excess (mingma: rev 0e54b164a8d8acf09aca8712116bf7a554cb4846)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #638 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/638/ ) HDFS-9314 . Improve BlockPlacementPolicyDefault's picking of excess (mingma: rev 0e54b164a8d8acf09aca8712116bf7a554cb4846) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2576 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2576/)
          HDFS-9314. Improve BlockPlacementPolicyDefault's picking of excess (mingma: rev 0e54b164a8d8acf09aca8712116bf7a554cb4846)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2576 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2576/ ) HDFS-9314 . Improve BlockPlacementPolicyDefault's picking of excess (mingma: rev 0e54b164a8d8acf09aca8712116bf7a554cb4846) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
          Hide
          xiaochen Xiao Chen added a comment -

          As discussed in HDFS-8647, I'm providing branch2.6/2.7 patch for this JIRA. Zhe Zhang please help review, thanks.
          Backported from branch-2, and conflicts below:

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
            HDFS-9007 which is in branch-2 but not 2.6/2.7. Mostly due to the extra parameter of chooseReplicaToDelete. Resolved by directly copying HDFS-9414's branch-2 commit (which has 9007) to delete the extra param - it’s not used anyways, and this makes backporting easier. See conflicts below.) Also fixed branch2.6 for diamond operator, which is the only diff between 2.6 and 2.7 patches.
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
            Not exist.
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java
            Not exist.
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
            Due to BlockPlacementPolicyDefault#chooseReplicaToDelete. Applied HDFS-9414's branch-2 commit directly.
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
            Due to BlockPlacementPolicyDefault#chooseReplicaToDelete. Applied HDFS-9414's branch-2 commit directly.
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java
            Not exist.
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
            Due to BlockPlacementPolicyDefault#chooseReplicaToDelete. Applied HDFS-9414's branch-2 commit directly.
          Show
          xiaochen Xiao Chen added a comment - As discussed in HDFS-8647 , I'm providing branch2.6/2.7 patch for this JIRA. Zhe Zhang please help review, thanks. Backported from branch-2, and conflicts below: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java HDFS-9007 which is in branch-2 but not 2.6/2.7. Mostly due to the extra parameter of chooseReplicaToDelete. Resolved by directly copying HDFS-9414 's branch-2 commit (which has 9007) to delete the extra param - it’s not used anyways, and this makes backporting easier. See conflicts below.) Also fixed branch2.6 for diamond operator, which is the only diff between 2.6 and 2.7 patches. hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java Not exist. hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithUpgradeDomain.java Not exist. hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java Due to BlockPlacementPolicyDefault#chooseReplicaToDelete . Applied HDFS-9414 's branch-2 commit directly. hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java Due to BlockPlacementPolicyDefault#chooseReplicaToDelete . Applied HDFS-9414 's branch-2 commit directly. hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithUpgradeDomain.java Not exist. hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java Due to BlockPlacementPolicyDefault#chooseReplicaToDelete . Applied HDFS-9414 's branch-2 commit directly.
          Hide
          xiaochen Xiao Chen added a comment -

          Applied both patches locally and verified compile ok, touched tests pass.

          Show
          xiaochen Xiao Chen added a comment - Applied both patches locally and verified compile ok, touched tests pass.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Xiao! +1 on the 2.7 and 2.6 patches. I just committed them.

          Show
          zhz Zhe Zhang added a comment - Thanks Xiao! +1 on the 2.7 and 2.6 patches. I just committed them.
          Hide
          xiaochen Xiao Chen added a comment -

          Thank you Zhe, really appreciate it!

          Show
          xiaochen Xiao Chen added a comment - Thank you Zhe, really appreciate it!
          Hide
          vinayrpet Vinayakumar B added a comment -

          branch-2.6 compilation is broken. Please update the code to support JDK 1.6.

          [ERROR] /home/vinay/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java:[1022,56] error: diamond operator is not supported in -source 1.6
          Show
          vinayrpet Vinayakumar B added a comment - branch-2.6 compilation is broken. Please update the code to support JDK 1.6. [ERROR] /home/vinay/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java:[1022,56] error: diamond operator is not supported in -source 1.6
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Vinay for catching this! I just committed an addendum fix.

          Show
          zhz Zhe Zhang added a comment - Thanks Vinay for catching this! I just committed an addendum fix.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              mingma Ming Ma
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development