Details
-
Type:
Sub-task
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.8.0, 3.0.0-alpha1
-
Component/s: None
-
Labels:None
-
Target Version/s:
Description
LightWeightHashSet requires fewer memory than java hashset.
Issue Links
- is related to
-
HDFS-11179
LightWeightHashSet can't remove blocks correctly which have a large number blockId
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
test failures of testNNClearsCommandsOnFailover* are related.
java.lang.UnsupportedOperationException: Remove is not supported. at org.apache.hadoop.hdfs.util.LightWeightHashSet$LinkedSetIterator.remove(LightWeightHashSet.java:565)
The remove of LightWeightHashSet$LinkedSetIterator is not implemented, actually it's easy to implement and will add it later.
Furthermore
public final Map<String, LightWeightLinkedSet<BlockInfo>> excessReplicateMap = new TreeMap<>();
we can use HashMap instead of TreeMap instead, since we don't need to sort. Will update the patch later.
Hi Yi Liu, do you have any estimates of the memory saved by using LightWeightHashSet?
Arpit Agarwal, sorry for late response
do you have any estimates of the memory saved by using LightWeightHashSet?
Yes, compared to java HashSet, there are two advantages from memory point of review:
- Java HashSet internally uses a HashMap, so there is one more reference (4 bytes) for each entry compared to LightWeightHashSet, so we can save 4 * size bytes of memory.
- In LightWeightHashSet, when elements become less, the size is shrinked a lot.
So we can see LightWeightHashSet is more better. The main issue is LightWeightHashSet#LinkedSetIterator doesn't support remove currently, it's easy to support it (similar to java HashSet). By the way, currently in Hadoop, we use LightWeightHashSet for all big objects required hash set except this one which needs to use remove.
Update a new patch. The new patch also improves excessReplicateMap, which we don't need sort, so no need to use a TreeMap.
public final Map<String, LightWeightLinkedSet<BlockInfo>> excessReplicateMap = - new TreeMap<>(); + new HashMap<>();
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| -1 | pre-patch | 15m 17s | Findbugs (version ) appears to be broken on trunk. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | javac | 7m 40s | There were no new javac warning messages. |
| +1 | javadoc | 9m 42s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. |
| +1 | checkstyle | 0m 36s | There were no new checkstyle issues. |
| -1 | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. |
| +1 | install | 1m 29s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 2m 28s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | native | 3m 3s | Pre-build of native portion |
| -1 | hdfs tests | 165m 14s | Tests failed in hadoop-hdfs. |
| 206m 25s |
| Reason | Tests |
|---|---|
| Failed unit tests | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12747948/HDFS-8792.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ddc867ce |
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/11874/artifact/patchprocess/whitespace.txt |
| hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11874/artifact/patchprocess/testrun_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/11874/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11874/console |
This message was automatically generated.
The test failure is not related.
Arpit Agarwal, could you help to review it? Thanks.
530 /** The starting modification for fail-fast. */
This JavaDoc comment is stale. It seems like it should be "the current modification epoch" or similar.
public final Map<String, LightWeightLinkedSet<BlockInfo>> excessReplicateMap = - new TreeMap<>(); + new HashMap<>();
Can you put this in a separate JIRA? It's less obvious to me that this is a win. Java HashMaps don't ever shrink, whereas TreeMap uses less memory when elements are removed.
Thanks Colin P. McCabe for the review.
This JavaDoc comment is stale. It seems like it should be "the current modification epoch" or similar.
Will update it in the new patch right now.
Can you put this in a separate JIRA? It's less obvious to me that this is a win. Java HashMaps don't ever shrink, whereas TreeMap uses less memory when elements are removed.
Sure, I will put it in a separate JIRA.
That's right HashMap don't ever shrink when elements are removed, but TreeMap entry needs to store more (memory) references (left, right, parent) than HashMap entry (only one reference next), even when there is element removing and cause some entry empty, the empty HashMap entry is just a null reference (4 bytes), so they are close at this point. On the other hand, the key of excessReplicateMap is datanode uuid, so the entries number is almost fixed, so HashMap memory is good than TreeMap memory in this case. I think the most important is the search/insert/remove performance, HashMap is absolutely better than TreeMap. Because we don't need to sort, we should use HashMap instead of TreeMap.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 17m 24s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | javac | 7m 40s | There were no new javac warning messages. |
| +1 | javadoc | 9m 44s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
| -1 | checkstyle | 1m 25s | The applied patch generated 2 new checkstyle issues (total was 187, now 188). |
| -1 | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. |
| +1 | install | 1m 20s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 2m 33s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | native | 3m 1s | Pre-build of native portion |
| -1 | hdfs tests | 161m 36s | Tests failed in hadoop-hdfs. |
| 205m 42s |
| Reason | Tests |
|---|---|
| Failed unit tests | hadoop.hdfs.server.namenode.ha.TestStandbyIsHot |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12748975/HDFS-8792.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f59612e |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/11914/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt |
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/11914/artifact/patchprocess/whitespace.txt |
| hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11914/artifact/patchprocess/testrun_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/11914/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11914/console |
This message was automatically generated.
Thanks for this, Yi Liu. +1. Test failure is unrelated. I will fix the lines which end in whitespace by using git apply --whitespace=fix.
FAILURE: Integrated in Hadoop-trunk-Commit #8314 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8314/)
HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
Thanks Colin P. McCabe for the review and commit!
FAILURE: Integrated in Hadoop-Yarn-trunk #1021 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1021/)
HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c)
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #291 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/291/)
HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #288 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/288/)
HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2237 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2237/)
HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
FAILURE: Integrated in Hadoop-Hdfs-trunk #2218 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2218/)
HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #280 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/280/)
HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) (cmccabe: rev c77bd6af16cbc26f88a2c6d8220db83a3e1caa2c)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
This seems to have a bug. Please see HDFS-11179.
This message was automatically generated.