[HDFS-8966] Separate the lock used in namespace and block management layer

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Target Version/s: HDFS-8966
• Assignee: Haohui Mai
• Reporter: Haohui Mai
• Created: 27/Aug/15 01:37
• Updated: 04/Apr/18 09:24

Description

Currently the namespace and the block management layer share one giant lock. One consequence that we have seen more and more often is that the namespace hangs due to excessive activities from the block management layer. For example, the NN might take a couple hundred milliseconds to handle a large block report. Because the NN holds the write lock during processing the block report, all namespace requests are paused. In production we have seen these lock contentions cause long latencies and instabilities in the cluster.

This umbrella jira proposes to separate the lock used by namespace and the block management layer.

Sub-Tasks

• Store INodeId instead of the INodeFile object in BlockInfoContiguous: Resolved
• Create a BlockManagerLock class to represent the lock used in the BlockManager: Resolved
• Convert BlockInfoUnderConstruction as a feature: Resolved
• Move replication factor into individual blocks: Resolved
• Remove unnecessary block replacement in INodeFile: Resolved
• Move replication queues related methods in FSNamesystem to BlockManager: Resolved
• Move the safemode block count into BlockManager: Resolved
• Code cleanup for DatanodeDescriptor and HeartbeatManager: Resolved
• Make DatanodeInfo thread safe: Patch Available
• Code cleanup for DatanodeManager: Resolved
• Move block replication logic from BlockManager to a new class ReplicationManager: Resolved
• Move code that tracks blocks with future generation stamps to BlockManagerSafeMode: Resolved
• Move BlockIdManager from FSNamesystem to BlockManager: Resolved
• Avoid unnecessary getBlockCollection calls in BlockManager: Resolved
• Make multi-thread access class BlockInfoContiguous thread safe: In Progress