[HDFS-16663] EC: Allow block reconstruction pending timeout refreshable to increase decommission performance

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: ec, namenode
• Labels: pull-request-available
• Assignee: caozhiqiang
• Reporter: caozhiqiang
• Created: 16/Jul/22 09:07
• Updated: 24/Jan/24 07:25
• Resolved: 12/Sep/22 02:47

Description

In HDFS-16613, increase the value of dfs.namenode.replication.max-streams-hard-limit would maximize the IO performance of the decommissioning DN, which has a lot of EC blocks. Besides this, we also need to decrease the value of dfs.namenode.reconstruction.pending.timeout-sec, default is 5 minutes, to shorten the interval time for checking pendingReconstructions. Or the decommissioning node would be idle to wait for copy tasks in most of this 5 minutes.

In decommission progress, we may need to reconfigure these 2 parameters several times. In HDFS-14560, the dfs.namenode.replication.max-streams-hard-limit can already be reconfigured dynamically without namenode restart. And the dfs.namenode.reconstruction.pending.timeout-sec parameter also need to be reconfigured dynamically. 

 

Issue Links

• relates to: HDFS-14560 Allow block replication parameters to be refreshable
• relates to: HDFS-16613 EC: Improve performance of decommissioning dn with many ec blocks
• links to: GitHub Pull Request #4567