[HADOOP-10610] Upgrade S3n fs.s3.buffer.dir to support multi directories

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.6.0
• Component/s: fs/s3
• Target Version/s: 2.6.0
• Hadoop Flags: Reviewed
• Assignee: Theodore michael Malaska
• Reporter: Theodore michael Malaska
• Created: 13/May/14 14:28
• Updated: 22/Aug/16 05:00
• Resolved: 17/Jul/14 21:58

Description

fs.s3.buffer.dir defines the tmp folder where files will be written to before getting sent to S3. Right now this is limited to a single folder which causes to major issues.

1. You need a drive with enough space to store all the tmp files at once
2. You are limited to the IO speeds of a single drive

This solution will resolve both and has been tested to increase the S3 write speed by 2.5x with 10 mappers on hs1.

Issue Links

• is cloned by: HADOOP-13530 Upgrade S3 fs.s3.buffer.dir to support multi directories