[HDFS-11786] Add support to make copyFromLocal multi threaded

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-beta1
• Component/s: hdfs
• Target Version/s: 3.0.0-beta1
• Hadoop Flags: Reviewed
• Assignee: Mukul Kumar Singh
• Reporter: Mukul Kumar Singh
• Created: 09/May/17 18:53
• Updated: 10/Aug/17 05:02
• Resolved: 16/Jul/17 18:21

Description

CopyFromLocal/Put is not currently multithreaded.

In case, where there are multiple files which need to be uploaded to the hdfs, a single thread reads the file and then copies the data to the cluster.

This copy to hdfs can be made faster by uploading multiple files in parallel.

I am attaching the initial patch so that I can get some initial feedback.

Issue Links

• breaks: HADOOP-14752 TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky
• is related to: HADOOP-14698 Make copyFromLocal's -t option available for put as well