Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8793 Improve BlockManager memory and performance
  3. HDFS-8792

BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      LightWeightHashSet requires fewer memory than java hashset.

      1. HDFS-8792.001.patch
        2 kB
        Yi Liu
      2. HDFS-8792.002.patch
        6 kB
        Yi Liu
      3. HDFS-8792.003.patch
        6 kB
        Yi Liu

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 20m 20s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 10m 44s There were no new javac warning messages.
          +1 javadoc 13m 31s There were no new javadoc warning messages.
          +1 release audit 0m 34s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 56s The applied patch generated 2 new checkstyle issues (total was 171, now 172).
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 47s mvn install still works.
          +1 eclipse:eclipse 0m 45s The patch built with eclipse:eclipse.
          +1 findbugs 3m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 40s Pre-build of native portion
          -1 hdfs tests 161m 4s Tests failed in hadoop-hdfs.
              217m 43s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestSetrepDecreasing
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
            hadoop.hdfs.TestLeaseRecovery2
            hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
            hadoop.hdfs.server.namenode.ha.TestDNFencing



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12745785/HDFS-8792.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / ee36f4f
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11737/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11737/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11737/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11737/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11737/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 20s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 10m 44s There were no new javac warning messages. +1 javadoc 13m 31s There were no new javadoc warning messages. +1 release audit 0m 34s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 56s The applied patch generated 2 new checkstyle issues (total was 171, now 172). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 47s mvn install still works. +1 eclipse:eclipse 0m 45s The patch built with eclipse:eclipse. +1 findbugs 3m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 40s Pre-build of native portion -1 hdfs tests 161m 4s Tests failed in hadoop-hdfs.     217m 43s   Reason Tests Failed unit tests hadoop.hdfs.TestSetrepDecreasing   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks   hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   hadoop.hdfs.server.namenode.ha.TestDNFencing Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12745785/HDFS-8792.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ee36f4f checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11737/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11737/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11737/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11737/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11737/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment -

          test failures of testNNClearsCommandsOnFailover* are related.

          java.lang.UnsupportedOperationException: Remove is not supported.
          	at org.apache.hadoop.hdfs.util.LightWeightHashSet$LinkedSetIterator.remove(LightWeightHashSet.java:565)
          

          The remove of LightWeightHashSet$LinkedSetIterator is not implemented, actually it's easy to implement and will add it later.

          Furthermore

          public final Map<String, LightWeightLinkedSet<BlockInfo>> excessReplicateMap =
              new TreeMap<>();
          

          we can use HashMap instead of TreeMap instead, since we don't need to sort. Will update the patch later.

          Show
          hitliuyi Yi Liu added a comment - test failures of testNNClearsCommandsOnFailover* are related. java.lang.UnsupportedOperationException: Remove is not supported. at org.apache.hadoop.hdfs.util.LightWeightHashSet$LinkedSetIterator.remove(LightWeightHashSet.java:565) The remove of LightWeightHashSet$LinkedSetIterator is not implemented, actually it's easy to implement and will add it later. Furthermore public final Map< String , LightWeightLinkedSet<BlockInfo>> excessReplicateMap = new TreeMap<>(); we can use HashMap instead of TreeMap instead, since we don't need to sort. Will update the patch later.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Yi Liu, do you have any estimates of the memory saved by using LightWeightHashSet?

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Yi Liu , do you have any estimates of the memory saved by using LightWeightHashSet?
          Hide
          hitliuyi Yi Liu added a comment -

          Arpit Agarwal, sorry for late response

          do you have any estimates of the memory saved by using LightWeightHashSet?

          Yes, compared to java HashSet, there are two advantages from memory point of review:

          1. Java HashSet internally uses a HashMap, so there is one more reference (4 bytes) for each entry compared to LightWeightHashSet, so we can save 4 * size bytes of memory.
          2. In LightWeightHashSet, when elements become less, the size is shrinked a lot.

          So we can see LightWeightHashSet is more better. The main issue is LightWeightHashSet#LinkedSetIterator doesn't support remove currently, it's easy to support it (similar to java HashSet). By the way, currently in Hadoop, we use LightWeightHashSet for all big objects required hash set except this one which needs to use remove.

          Show
          hitliuyi Yi Liu added a comment - Arpit Agarwal , sorry for late response do you have any estimates of the memory saved by using LightWeightHashSet? Yes, compared to java HashSet , there are two advantages from memory point of review: Java HashSet internally uses a HashMap , so there is one more reference (4 bytes) for each entry compared to LightWeightHashSet , so we can save 4 * size bytes of memory. In LightWeightHashSet , when elements become less, the size is shrinked a lot. So we can see LightWeightHashSet is more better. The main issue is LightWeightHashSet#LinkedSetIterator doesn't support remove currently, it's easy to support it (similar to java HashSet). By the way, currently in Hadoop, we use LightWeightHashSet for all big objects required hash set except this one which needs to use remove .
          Hide
          hitliuyi Yi Liu added a comment -

          Update a new patch. The new patch also improves excessReplicateMap, which we don't need sort, so no need to use a TreeMap.

             public final Map<String, LightWeightLinkedSet<BlockInfo>> excessReplicateMap =
          -    new TreeMap<>();
          +    new HashMap<>();
          
          Show
          hitliuyi Yi Liu added a comment - Update a new patch. The new patch also improves excessReplicateMap , which we don't need sort , so no need to use a TreeMap . public final Map< String , LightWeightLinkedSet<BlockInfo>> excessReplicateMap = - new TreeMap<>(); + new HashMap<>();
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 15m 17s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 40s There were no new javac warning messages.
          +1 javadoc 9m 42s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 36s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 29s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 3s Pre-build of native portion
          -1 hdfs tests 165m 14s Tests failed in hadoop-hdfs.
              206m 25s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12747948/HDFS-8792.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / ddc867ce
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11874/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11874/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11874/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11874/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 17s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 40s There were no new javac warning messages. +1 javadoc 9m 42s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 36s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 3s Pre-build of native portion -1 hdfs tests 165m 14s Tests failed in hadoop-hdfs.     206m 25s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747948/HDFS-8792.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ddc867ce whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11874/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11874/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11874/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11874/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment -

          The test failure is not related.

          Show
          hitliuyi Yi Liu added a comment - The test failure is not related.
          Hide
          hitliuyi Yi Liu added a comment -

          Arpit Agarwal, could you help to review it? Thanks.

          Show
          hitliuyi Yi Liu added a comment - Arpit Agarwal , could you help to review it? Thanks.
          Hide
          cmccabe Colin P. McCabe added a comment -
          530	    /** The starting modification for fail-fast. */
          

          This JavaDoc comment is stale. It seems like it should be "the current modification epoch" or similar.

             public final Map<String, LightWeightLinkedSet<BlockInfo>> excessReplicateMap =
          -    new TreeMap<>();
          +    new HashMap<>();
          

          Can you put this in a separate JIRA? It's less obvious to me that this is a win. Java HashMaps don't ever shrink, whereas TreeMap uses less memory when elements are removed.

          Show
          cmccabe Colin P. McCabe added a comment - 530 /** The starting modification for fail-fast. */ This JavaDoc comment is stale. It seems like it should be "the current modification epoch" or similar. public final Map< String , LightWeightLinkedSet<BlockInfo>> excessReplicateMap = - new TreeMap<>(); + new HashMap<>(); Can you put this in a separate JIRA? It's less obvious to me that this is a win. Java HashMaps don't ever shrink, whereas TreeMap uses less memory when elements are removed.
          Hide
          hitliuyi Yi Liu added a comment - - edited

          Thanks Colin P. McCabe for the review.

          This JavaDoc comment is stale. It seems like it should be "the current modification epoch" or similar.

          Will update it in the new patch right now.

          Can you put this in a separate JIRA? It's less obvious to me that this is a win. Java HashMaps don't ever shrink, whereas TreeMap uses less memory when elements are removed.

          Sure, I will put it in a separate JIRA.
          That's right HashMap don't ever shrink when elements are removed, but TreeMap entry needs to store more (memory) references (left, right, parent) than HashMap entry (only one reference next), even when there is element removing and cause some entry empty, the empty HashMap entry is just a null reference (4 bytes), so they are close at this point. On the other hand, the key of excessReplicateMap is datanode uuid, so the entries number is almost fixed, so HashMap memory is good than TreeMap memory in this case. I think the most important is the search/insert/remove performance, HashMap is absolutely better than TreeMap. Because we don't need to sort, we should use HashMap instead of TreeMap.

          Show
          hitliuyi Yi Liu added a comment - - edited Thanks Colin P. McCabe for the review. This JavaDoc comment is stale. It seems like it should be "the current modification epoch" or similar. Will update it in the new patch right now. Can you put this in a separate JIRA? It's less obvious to me that this is a win. Java HashMaps don't ever shrink, whereas TreeMap uses less memory when elements are removed. Sure, I will put it in a separate JIRA. That's right HashMap don't ever shrink when elements are removed, but TreeMap entry needs to store more (memory) references (left, right, parent) than HashMap entry (only one reference next), even when there is element removing and cause some entry empty, the empty HashMap entry is just a null reference (4 bytes), so they are close at this point. On the other hand, the key of excessReplicateMap is datanode uuid, so the entries number is almost fixed, so HashMap memory is good than TreeMap memory in this case. I think the most important is the search/insert/remove performance, HashMap is absolutely better than TreeMap. Because we don't need to sort, we should use HashMap instead of TreeMap.
          Hide
          hitliuyi Yi Liu added a comment -

          Update the new patch.

          Show
          hitliuyi Yi Liu added a comment - Update the new patch.
          Hide
          hitliuyi Yi Liu added a comment -

          I move BlockManager#excessReplicateMap to HDFS-8862.

          Show
          hitliuyi Yi Liu added a comment - I move BlockManager#excessReplicateMap to HDFS-8862 .
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 24s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 40s There were no new javac warning messages.
          +1 javadoc 9m 44s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 25s The applied patch generated 2 new checkstyle issues (total was 187, now 188).
          -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 2m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 1s Pre-build of native portion
          -1 hdfs tests 161m 36s Tests failed in hadoop-hdfs.
              205m 42s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyIsHot



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12748975/HDFS-8792.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f59612e
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11914/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11914/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11914/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11914/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11914/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 24s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 40s There were no new javac warning messages. +1 javadoc 9m 44s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 25s The applied patch generated 2 new checkstyle issues (total was 187, now 188). -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 1s Pre-build of native portion -1 hdfs tests 161m 36s Tests failed in hadoop-hdfs.     205m 42s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyIsHot Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748975/HDFS-8792.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f59612e checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11914/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11914/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11914/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11914/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11914/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks for this, Yi Liu. +1. Test failure is unrelated. I will fix the lines which end in whitespace by using git apply --whitespace=fix.

          Show
          cmccabe Colin P. McCabe added a comment - Thanks for this, Yi Liu . +1. Test failure is unrelated. I will fix the lines which end in whitespace by using git apply --whitespace=fix .
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8314 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8314/)
          HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8314 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8314/ ) HDFS-8792 . BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          Hide
          hitliuyi Yi Liu added a comment -

          Thanks Colin P. McCabe for the review and commit!

          Show
          hitliuyi Yi Liu added a comment - Thanks Colin P. McCabe for the review and commit!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1021 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1021/)
          HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1021 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1021/ ) HDFS-8792 . BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #291 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/291/)
          HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #291 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/291/ ) HDFS-8792 . BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #288 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/288/)
          HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #288 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/288/ ) HDFS-8792 . BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2237 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2237/)
          HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2237 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2237/ ) HDFS-8792 . BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2218 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2218/)
          HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2218 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2218/ ) HDFS-8792 . BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #280 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/280/)
          HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #280 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/280/ ) HDFS-8792 . BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          Hide
          tasanuma0829 Takanobu Asanuma added a comment -

          This seems to have a bug. Please see HDFS-11179.

          Show
          tasanuma0829 Takanobu Asanuma added a comment - This seems to have a bug. Please see HDFS-11179 .

            People

            • Assignee:
              hitliuyi Yi Liu
              Reporter:
              hitliuyi Yi Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development