[HDFS-7244] Reduce Namenode memory using Flyweight pattern

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: namenode
• Assignee: Unassigned
• Reporter: Amir Langer
• Created: 14/Oct/14 11:30
• Updated: 12/Nov/14 07:15

Description

Using the flyweight pattern can dramatically reduce memory usage in the Namenode. The pattern also abstracts the actual storage type and allows the decision of whether it is off-heap or not and what is the serialisation mechanism to be configured per deployment.

The idea is to move all BlockInfo data (as a first step) to this storage using the Flyweight pattern. The cost to doing it will be in higher latency when accessing/modifying a block. The idea is that this will be offset with a reduction in memory and in the case of off-heap, a dramatic reduction in memory (effectively, memory used for BlockInfo would reduce to a very small constant value).
This reduction will also have an huge impact on the latency as GC pauses will be reduced considerably and may even end up with better latency results than the original code.

I wrote a stand-alone project as a proof of concept, to show the pattern, the data structure we can use and what will be the performance costs of this approach.

see Slab
and Slab performance results.

Slab abstracts the storage, gives several storage implementations and implements the flyweight pattern for the application (Namenode in our case).
The stages to incorporate Slab into the Namenode is outlined in the sub-tasks JIRAs.

Issue Links

• relates to: HDFS-6709 Implement off-heap data structures for NameNode and other HDFS memory optimization
• relates to: HDFS-6658 Namenode memory optimization - Block replicas list

Sub-Tasks

• Introduce Slab code in HDFS: Open
• Use ids for DatanodeStorageInfo in the BlockInfo triplets - HDFS 6660: Open
• Use ids for Block collection in Block: Open
• Use ids for blocks in InodeFile: Open
• Define block slabs per replication factor and initialise them in advance (inc. size config): Open
• Store blocks in slabs rather than a Map inside BlocksMap: Open