[HDFS-17216] When distcp handle the small files, the bandwidth parameter will be invalid, resulting in serious overspeed behavior

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.4
• Fix Version/s: 3.5.0
• Component/s: distcp
• Labels: pull-request-available
• Assignee: xiaojunxiang
• Reporter: xiaojunxiang
• Created: 02/Oct/23 01:21
• Updated: 03/Apr/24 17:33
• Resolved: 28/Mar/24 14:49

Description

When distcp copies small files (file size slightly smaller than the bandwidth), the throbber only starts to throb after 1 second, and the throttled is specific to a single file. so the throbber becomes invalid, causing distcp to fill the cluster bandwidth and crush production traffic, which is a terrible thing.

Also, it takes time for files to set up the IO pipeline, so you shouldn't test with very small files, which will slow the transfer, especially as bandwidth kicks in, which will amplify the impact of small files on the rate

Issue Links

• links to: GitHub Pull Request #6138