[HADOOP-14766] Cloudup: an object store high performance dfs put command

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 2.8.1
• Component/s: fs, fs/azure, fs/s3
• Target Version/s: 3.4.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 12/Aug/17 17:05
• Updated: 21/Jan/21 11:53
• Resolved: 21/Jan/21 11:53

Description

hdfs put local s3a://path is suboptimal as it treewalks down down the source tree then, sequentially, copies up the file through copying the file (opened as a stream) contents to a buffer, writes that to the dest file, repeats.

For S3A that hurts because

• it;s doing the upload inefficiently: the file can be uploaded just by handling the pathname to the AWS xter manager
• it is doing it sequentially, when some parallelised upload would work.
• as the ordering of the files to upload is a recursive treewalk, it doesn't spread the upload across multiple shards.

Better:

• build the list of files to upload
• upload in parallel, picking entries from the list at random and spreading across a pool of uploaders
• upload straight from local file (copyFromLocalFile()
• track IO load (files created/second) to estimate risk of throttling.

Issue Links

• depends upon: HADOOP-14432 S3A copyFromLocalFile to be robust, tested
• is related to: HADOOP-14767 WASB to implement copyFromLocalFile()