Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9313

Possible NullPointerException in BlockManager if no excess replica can be chosen

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-8647 makes it easier to reason about various block placement scenarios. Here is one possible case where BlockManager won't be able to find the excess replica to delete: when storage policy changes around the same time balancer moves the block. When this happens, it will cause NullPointerException.

      java.lang.NullPointerException
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
      

      Note that it isn't found in any production clusters. Instead, it is found from new unit tests. In addition, the issue has been there before HDFS-8647.

      1. HDFS-9313.branch26.patch
        4 kB
        Xiao Chen
      2. HDFS-9313.branch27.patch
        4 kB
        Xiao Chen
      3. HDFS-9313.patch
        5 kB
        Ming Ma
      4. HDFS-9313-2.patch
        5 kB
        Ming Ma

        Issue Links

          Activity

          Hide
          mingma Ming Ma added a comment -

          Here is the patch that illustrates the scenario. It is better to guard against this.

          In addition, for this specific test scenario, BlockPlacementPolicyDefault should have been able to delete excessSSD. We can fix it separately.

          Show
          mingma Ming Ma added a comment - Here is the patch that illustrates the scenario. It is better to guard against this. In addition, for this specific test scenario, BlockPlacementPolicyDefault should have been able to delete excessSSD. We can fix it separately.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for filing and working on this, Ming Ma. I think the patch makes sense to me. The warning is much better than a NPE.

          +    // no replica can't be chosen as the excessive replica as
          

          Do you mean "no replica can be chosen as the excessive replica as"?

          Show
          liuml07 Mingliang Liu added a comment - Thanks for filing and working on this, Ming Ma . I think the patch makes sense to me. The warning is much better than a NPE. + // no replica can't be chosen as the excessive replica as Do you mean "no replica can be chosen as the excessive replica as"?
          Hide
          walter.k.su Walter Su added a comment -

          I'm ok that adding a null check. However, I don't think it's enough to address the scenario here, In the test case, you added 1 SSD + 3 DISKs. As you said in the patch,

          1040	    // In this case,
          1041	    // no replica can't be chosen as the excessive replica as
          1042	    // chooseReplicasToDelete only considers storages[4] and storages[5] that
          1043	    // are the same rack. But neither's storage type is SSD.
          

          If we choose nothing, the replica on SSD won't be deleted. And I remember, Mover won't do it neither, since the existings contains the expected.

          Instead of choosing nothing, we should choose the SSD, since the remaining 3 DISKs are already on enough racks.

          Show
          walter.k.su Walter Su added a comment - I'm ok that adding a null check. However, I don't think it's enough to address the scenario here, In the test case, you added 1 SSD + 3 DISKs. As you said in the patch, 1040 // In this case , 1041 // no replica can't be chosen as the excessive replica as 1042 // chooseReplicasToDelete only considers storages[4] and storages[5] that 1043 // are the same rack. But neither's storage type is SSD. If we choose nothing, the replica on SSD won't be deleted. And I remember, Mover won't do it neither, since the existings contains the expected. Instead of choosing nothing, we should choose the SSD, since the remaining 3 DISKs are already on enough racks.
          Hide
          liuml07 Mingliang Liu added a comment -

          Will this be covered by HDFS-9314?

          Show
          liuml07 Mingliang Liu added a comment - Will this be covered by HDFS-9314 ?
          Hide
          walter.k.su Walter Su added a comment -

          yes.

          Show
          walter.k.su Walter Su added a comment - yes.
          Hide
          mingma Ming Ma added a comment -

          The question is, even after HDFS-9314 is fixed we might still run into other scenarios where no excess replica can't be picked. It becomes more likely we continue to add new block placement policies; thus adding null check is more of defense in depth.

          Show
          mingma Ming Ma added a comment - The question is, even after HDFS-9314 is fixed we might still run into other scenarios where no excess replica can't be picked. It becomes more likely we continue to add new block placement policies; thus adding null check is more of defense in depth.
          Hide
          liuml07 Mingliang Liu added a comment -

          Agreed.

          +1 (non-binding) pending on Jenkins.

          Show
          liuml07 Mingliang Liu added a comment - Agreed. +1 (non-binding) pending on Jenkins.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks Ming Ma for reporting this.. I too agree.. +1 (non-binding)

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks Ming Ma for reporting this.. I too agree.. +1 (non-binding)
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 25m 4s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 10m 47s There were no new javac warning messages.
          +1 javadoc 14m 7s There were no new javadoc warning messages.
          +1 release audit 0m 31s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 3s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 2m 5s mvn install still works.
          +1 eclipse:eclipse 0m 44s The patch built with eclipse:eclipse.
          +1 findbugs 3m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 37s Pre-build of native portion
          -1 hdfs tests 67m 32s Tests failed in hadoop-hdfs.
              130m 8s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestRead
            hadoop.hdfs.server.namenode.TestStorageRestore
            hadoop.hdfs.server.namenode.TestParallelImageWrite
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
            hadoop.hdfs.server.namenode.TestNameNodeMXBean
            hadoop.hdfs.security.token.block.TestBlockToken
            hadoop.hdfs.server.namenode.TestFSImage
            hadoop.hdfs.server.namenode.TestAddBlockRetry
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
            hadoop.hdfs.server.namenode.TestXAttrConfigFlag
            hadoop.hdfs.security.TestDelegationToken
          Timed out tests org.apache.hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12768868/HDFS-9313.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 96677be
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13212/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13212/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13212/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13212/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 25m 4s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 10m 47s There were no new javac warning messages. +1 javadoc 14m 7s There were no new javadoc warning messages. +1 release audit 0m 31s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 3s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 2m 5s mvn install still works. +1 eclipse:eclipse 0m 44s The patch built with eclipse:eclipse. +1 findbugs 3m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 37s Pre-build of native portion -1 hdfs tests 67m 32s Tests failed in hadoop-hdfs.     130m 8s   Reason Tests Failed unit tests hadoop.hdfs.TestRead   hadoop.hdfs.server.namenode.TestStorageRestore   hadoop.hdfs.server.namenode.TestParallelImageWrite   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.namenode.TestProcessCorruptBlocks   hadoop.hdfs.server.namenode.TestNameNodeMXBean   hadoop.hdfs.security.token.block.TestBlockToken   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.server.namenode.TestAddBlockRetry   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.server.namenode.TestNameNodeResourceChecker   hadoop.hdfs.server.namenode.TestXAttrConfigFlag   hadoop.hdfs.security.TestDelegationToken Timed out tests org.apache.hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768868/HDFS-9313.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 96677be Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13212/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13212/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13212/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13212/console This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          New patch to fix the issue Mingliang pointed out. Thanks Mingliang, Water, Brahma.

          Show
          mingma Ming Ma added a comment - New patch to fix the issue Mingliang pointed out. Thanks Mingliang, Water, Brahma.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 21m 6s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 9m 11s There were no new javac warning messages.
          +1 javadoc 12m 3s There were no new javadoc warning messages.
          +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 40s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 48s mvn install still works.
          +1 eclipse:eclipse 0m 40s The patch built with eclipse:eclipse.
          +1 findbugs 2m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 42s Pre-build of native portion
          -1 hdfs tests 55m 9s Tests failed in hadoop-hdfs.
              108m 46s  



          Reason Tests
          Failed unit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12768993/HDFS-9313-2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / bcb2386
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13225/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13225/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13225/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13225/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 21m 6s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 9m 11s There were no new javac warning messages. +1 javadoc 12m 3s There were no new javadoc warning messages. +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 40s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 48s mvn install still works. +1 eclipse:eclipse 0m 40s The patch built with eclipse:eclipse. +1 findbugs 2m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 42s Pre-build of native portion -1 hdfs tests 55m 9s Tests failed in hadoop-hdfs.     108m 46s   Reason Tests Failed unit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768993/HDFS-9313-2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / bcb2386 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13225/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13225/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13225/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13225/console This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Ming for the fix. I haven't dived into full details of the two chooseReplicaToDelete methods, but a quick question is why we should break instead of continue when seeing a null value?

          Show
          zhz Zhe Zhang added a comment - Thanks Ming for the fix. I haven't dived into full details of the two chooseReplicaToDelete methods, but a quick question is why we should break instead of continue when seeing a null value?
          Hide
          mingma Ming Ma added a comment -

          Thanks Zhe Zhang. If it continues, given no state has changed or no alternative approach is taken, it will just keep looping.

          Show
          mingma Ming Ma added a comment - Thanks Zhe Zhang . If it continues, given no state has changed or no alternative approach is taken, it will just keep looping.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Ming for explaining this. I agree break is the right logic here. +1 on the latest patch.

          Show
          zhz Zhe Zhang added a comment - Thanks Ming for explaining this. I agree break is the right logic here. +1 on the latest patch.
          Hide
          mingma Ming Ma added a comment -

          Committed to trunk and branch-2. Thanks Zhe Zhang, Mingliang Liu, Walter Su and Brahma Reddy Battula.

          Show
          mingma Ming Ma added a comment - Committed to trunk and branch-2. Thanks Zhe Zhang , Mingliang Liu , Walter Su and Brahma Reddy Battula .
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #620 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/620/)
          HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma: rev d565480da2f646b40c3180e1ccb2935c9863dfef)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #620 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/620/ ) HDFS-9313 . Possible NullPointerException in BlockManager if no excess (mingma: rev d565480da2f646b40c3180e1ccb2935c9863dfef) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8746 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8746/)
          HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma: rev d565480da2f646b40c3180e1ccb2935c9863dfef)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8746 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8746/ ) HDFS-9313 . Possible NullPointerException in BlockManager if no excess (mingma: rev d565480da2f646b40c3180e1ccb2935c9863dfef) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2561 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2561/)
          HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma: rev d565480da2f646b40c3180e1ccb2935c9863dfef)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2561 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2561/ ) HDFS-9313 . Possible NullPointerException in BlockManager if no excess (mingma: rev d565480da2f646b40c3180e1ccb2935c9863dfef) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1354 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1354/)
          HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma: rev d565480da2f646b40c3180e1ccb2935c9863dfef)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1354 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1354/ ) HDFS-9313 . Possible NullPointerException in BlockManager if no excess (mingma: rev d565480da2f646b40c3180e1ccb2935c9863dfef) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #631 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/631/)
          HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma: rev d565480da2f646b40c3180e1ccb2935c9863dfef)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #631 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/631/ ) HDFS-9313 . Possible NullPointerException in BlockManager if no excess (mingma: rev d565480da2f646b40c3180e1ccb2935c9863dfef) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #566 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/566/)
          HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma: rev d565480da2f646b40c3180e1ccb2935c9863dfef)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #566 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/566/ ) HDFS-9313 . Possible NullPointerException in BlockManager if no excess (mingma: rev d565480da2f646b40c3180e1ccb2935c9863dfef) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2502 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2502/)
          HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma: rev d565480da2f646b40c3180e1ccb2935c9863dfef)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2502 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2502/ ) HDFS-9313 . Possible NullPointerException in BlockManager if no excess (mingma: rev d565480da2f646b40c3180e1ccb2935c9863dfef) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
          Hide
          xiaochen Xiao Chen added a comment -

          As discussed in HDFS-8647, I'm providing branch2.6/2.7 patch for this JIRA.
          Cherry-picking from branch2, the backport was clean. But I had to fix a compile error about HdfsServerConstants due to file change. Additionally I removed the logging change, in order for the code to run. (Otherwise error at run time: (java.lang.ClassCastException: org.slf4j.impl.Log4jLoggerAdapter cannot be cast to org.apache.commons.logging.impl.Log4JLogger)).
          Locally tested both patches compile and passes TestReplicationPolicy.

          Show
          xiaochen Xiao Chen added a comment - As discussed in HDFS-8647 , I'm providing branch2.6/2.7 patch for this JIRA. Cherry-picking from branch2, the backport was clean. But I had to fix a compile error about HdfsServerConstants due to file change. Additionally I removed the logging change, in order for the code to run. (Otherwise error at run time: (java.lang.ClassCastException: org.slf4j.impl.Log4jLoggerAdapter cannot be cast to org.apache.commons.logging.impl.Log4JLogger)). Locally tested both patches compile and passes TestReplicationPolicy.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Xiao! +1 on the patches. I just committed to branch-2.7 and branch-2.6.

          Show
          zhz Zhe Zhang added a comment - Thanks Xiao! +1 on the patches. I just committed to branch-2.7 and branch-2.6.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks again Zhe for the commit!

          Show
          xiaochen Xiao Chen added a comment - Thanks again Zhe for the commit!
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

            People

            • Assignee:
              mingma Ming Ma
              Reporter:
              mingma Ming Ma
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development