[HDFS-13183] Standby NameNode process getBlocks request to reduce Active load - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: New Feature
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 3.3.1, 3.4.0
Component/s: balancer & mover, namenode
Labels:
None

Release Note:

Hide
Enable balancer to redirect getBlocks request to a Standby Namenode, thus reducing the performance impact of balancer to the Active NameNode.

The feature is disabled by default. To enable it, configure the hdfs-site.xml of balancer:
dfs.ha.allow.stale.reads = true.

Show
Enable balancer to redirect getBlocks request to a Standby Namenode, thus reducing the performance impact of balancer to the Active NameNode. The feature is disabled by default. To enable it, configure the hdfs-site.xml of balancer: dfs.ha.allow.stale.reads = true.

Description

The performance of Active NameNode could be impact when Balancer requests #getBlocks, since query blocks of overly full DNs performance is extremely inefficient currently. The main reason is NameNodeRpcServer#getBlocks hold read lock for long time. In extreme case, all handlers of Active NameNode RPC server are occupied by one reader NameNodeRpcServer#getBlocks and other write operation calls, thus Active NameNode enter a state of false death for number of seconds even for minutes.
The similar performance concerns of Balancer have reported by ~~HDFS-9412~~, HDFS-7967, etc.

If Standby NameNode can shoulder #getBlocks heavy burden, it could speed up the progress of balancing and reduce performance impact to Active NameNode.