[HDFS-15253] Set default throttle value on dfs.image.transfer.bandwidthPerSec

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1, 3.4.0
• Fix Version/s: 3.3.1, 3.4.0
• Component/s: namenode
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Release Note: The configuration dfs.image.transfer.bandwidthPerSec which defines the maximum bandwidth available for fsimage transfer is changed from 0 (meaning no throttle at all) to 50MB/s.
• Assignee: Karthik Palanisamy
• Reporter: Karthik Palanisamy
• Created: 31/Mar/20 20:24
• Updated: 27/Jan/24 03:26
• Resolved: 07/Oct/20 16:45

Description

The default value dfs.image.transfer.bandwidthPerSec is set to 0 so it can use maximum available bandwidth for fsimage transfers during checkpoint. I think we should throttle this. Many users were experienced namenode failover when transferring large image size along with fsimage replication on dfs.namenode.name.dir. eg. >25Gb.  

Thought to set,

dfs.image.transfer.bandwidthPerSec=52428800. (50 MB/s)

dfs.namenode.checkpoint.txns=2000000 (Default is 1M, good to avoid frequent checkpoint. However, the default checkpoint runs every 6 hours once)

 

Issue Links

• links to: GitHub Pull Request #2366