[HADOOP-14086] Improve DistCp Speed for small files

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.6.5
• Component/s: tools/distcp
• Target Version/s: 2.6.6
• Assignee: Zheng Shao
• Reporter: Zheng Shao
• Created: 15/Feb/17 21:55
• Updated: 25/Sep/18 10:51

Description

When using distcp to copy lots of small files, NameNode naturally becomes a bottleneck.

The current distcp code did not optimize to reduce the NameNode calls. We should restructure the code to reduce the number of NameNode calls as much as possible to speed up the copy of small files.

Issue Links

• is depended upon by: HADOOP-15788 Improve Distcp for long-haul/cloud deployments