[HADOOP-15087] S3A to support writing directly to the destination dir without creating temp directory to avoid rename

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.0.0
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Yonger
• Created: 04/Dec/17 03:25
• Updated: 03/Apr/19 14:49
• Resolved: 03/Apr/19 14:49

Description

Rename in workloads like Teragen/Terasort who use Hadoop default outputcommitters really hurt performance a lot.
Stocator announce it doesn't create the temporary directories any all, and still preserves Hadoop's fault tolerance. I add a switch when creating file via integrating it's code into s3a, I got 5x performance gain in Teragen and 15% performance improvement in Terasort.

Issue Links

• duplicates: HADOOP-13786 Add S3A committers for zero-rename commits to S3 endpoints