[HDFS-14703] NameNode Fine-Grained Locking via Metadata Partitioning

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: hdfs, namenode
• Target Version/s: Fine-Grained Locking
• Assignee: Unassigned
• Reporter: Konstantin Shvachko
• Created: 06/Aug/19 01:36
• Updated: 10/Sep/21 10:46

Description

We target to enable fine-grained locking by splitting the in-memory namespace into multiple partitions each having a separate lock. Intended to improve performance of NameNode write operations.

Sub-Tasks

• [FGL] Remove redundant locking on NameNode.: Resolved
• [FGL] Introduce Latch Lock to replace Namesystem global lock.: Open
• [FGL] Introduce PartitionedGSet a new implementation of GSet.: Open
• [FGL] Introduce INode key.: Open
• [FGL] Implement Create File with FGL: Resolved
• [FGL] Fix the iterator for PartitionedGSet: Resolved
• [FGL] Add support for saving/loading an FS Image for PartitionedGSet: Resolved
• [FGL] Address permission related issues with File / Directory: Resolved
• [FGL] Fix FSImage loading issues on dynamic partitions: In Progress
• [FGL] Implement Append & Rename APIs with FGL: In Progress
• [FGL] Implement Delete API with FGL: In Progress
• [FGL]Configurable INodeMap#NAMESPACE_KEY_DEPTH&NUM_RANGES_STATIC: In Progress
• [FGL]Improve safer PartitionedGSet#size: In Progress
• [FGL]Access to Create File is more secure: In Progress