[HDFS-13183] Standby NameNode process getBlocks request to reduce Active load

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.1, 3.4.0
• Component/s: balancer & mover, namenode

Release Note

Enable balancer to redirect getBlocks request to a Standby Namenode, thus reducing the performance impact of balancer to the Active NameNode.

The feature is disabled by default. To enable it, configure the hdfs-site.xml of balancer:
dfs.ha.allow.stale.reads = true.

• Assignee: Xiaoqiao He
• Reporter: Xiaoqiao He
• Created: 23/Feb/18 08:32
• Updated: 24/May/21 01:26
• Resolved: 28/May/20 07:39

Description

The performance of Active NameNode could be impact when Balancer requests #getBlocks, since query blocks of overly full DNs performance is extremely inefficient currently. The main reason is NameNodeRpcServer#getBlocks hold read lock for long time. In extreme case, all handlers of Active NameNode RPC server are occupied by one reader NameNodeRpcServer#getBlocks and other write operation calls, thus Active NameNode enter a state of false death for number of seconds even for minutes.
The similar performance concerns of Balancer have reported by HDFS-9412, HDFS-7967, etc.

If Standby NameNode can shoulder #getBlocks heavy burden, it could speed up the progress of balancing and reduce performance impact to Active NameNode.

Issue Links

• blocks: HDFS-15368 TestBalancerWithHANameNodes#testBalancerWithObserver failed occasionally
• is blocked by: HDFS-12976 Introduce ObserverReadProxyProvider
• relates to: HDFS-14162 Balancer should work with ObserverNode