Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Invalid
-
None
-
None
-
None
-
None
Description
Currently LightWeightHashSet and LightWeightLinkedSet are used in hdfs, there are two advantages compared to java HashSet: one is the entry requires fewer memory, another is it's shrinkable. In real cluster, hdfs is a long running service, and set may become large at some time and may become small after that, so shrinking the set when size hits the shrink threshold is necessary, it can improve the NN memory.
Same situation for map, some HashMap used in BlockManager (e.g., the hashmap in CorruptReplicasMap), it's better to be shrinkable.
I think it's worth to implement ShrinkableHashMap extends the java HashMap, for quick glance, seems few code is needed.