[HDFS-6658] Namenode memory optimization - Block replicas list

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.1
• Component/s: namenode
• Assignee: Daryn Sharp
• Reporter: Amir Langer
• Created: 10/Jul/14 11:07
• Updated: 06/Jan/17 21:58

Description

Part of the memory consumed by every BlockInfo object in the Namenode is a linked list of block references for every DatanodeStorageInfo (called "triplets").
We propose to change the way we store the list in memory.
Using primitive integer indexes instead of object references will reduce the memory needed for every block replica (when compressed oops is disabled) and in our new design the list overhead will be per DatanodeStorageInfo and not per block replica.

see attached design doc. for details and evaluation results.

Issue Links

• is related to: HDFS-7244 Reduce Namenode memory using Flyweight pattern
• relates to: HDFS-7836 BlockManager Scalability Improvements

Sub-Tasks

• Create a Block List: Patch Available
• Use int instead of object reference to DatanodeStorageInfo in BlockInfo triplets,: Patch Available
• Use BlockList instead of double linked list i.e BlockInfo triplets: Open