[HADOOP-10560] Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads

• Type: Improvement
• Status: Patch Available
• Priority: Minor
• Resolution: Unresolved
• Component/s: fs/s3
• Labels: BB2015-05-TBR performance
• Assignee: Theodore michael Malaska
• Reporter: Theodore michael Malaska
• Created: 01/May/14 17:35
• Updated: 07/May/15 09:20

Description

In NativeS3FileSystem if you do a copy of a directory it will copy all the files to the new location, but it will do this with one thread. Code is below. This jira will allow a configurable number of threads to be used to issue the copy commands to S3.
do

{ PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH, priorLastKey, true); for (FileMetadata file : listing.getFiles()) { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey + file.getKey().substring(srcKey.length())); }

priorLastKey = listing.getPriorLastKey();
} while (priorLastKey != null);