[HDFS-16846] EC: Only EC blocks should be effected by max-streams-hard-limit configuration

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: erasure-coding
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: caozhiqiang
• Reporter: caozhiqiang
• Created: 17/Nov/22 05:20
• Updated: 28/Jan/24 07:37
• Resolved: 29/Nov/22 01:53

Description

In HDFS-16613, the dfs.namenode.replication.max-streams-hard-limit configuration will only affect decommissioning DataNode, but will not distinguish between replication blocks and EC blocks. Even if DataNodes have only replication files, they will always generate high network traffic. So this configuration should only effect EC blocks.

Issue Links

• is related to: HDFS-16613 EC: Improve performance of decommissioning dn with many ec blocks
• links to: GitHub Pull Request #5143