[HADOOP-13530] Upgrade S3 fs.s3.buffer.dir to support multi directories

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 2.4.0
• Component/s: fs/s3
• Hadoop Flags: Reviewed
• Assignee: Theodore michael Malaska
• Reporter: Adrian Muraru
• Created: 22/Aug/16 05:00
• Updated: 22/Aug/16 08:50
• Resolved: 22/Aug/16 08:50

Description

fs.s3.buffer.dir defines the tmp folder where files will be written to before getting sent to S3. Right now this is limited to a single folder which causes to major issues.

1. You need a drive with enough space to store all the tmp files at once
2. You are limited to the IO speeds of a single drive

This is similar to HADOOP-10610 but applies to s3:// hadoop block fs.

Issue Links

• is a clone of: HADOOP-10610 Upgrade S3n fs.s3.buffer.dir to support multi directories