[HADOOP-13786] Add S3A committers for zero-rename commits to S3 endpoints

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-beta1
• Fix Version/s: 3.1.0
• Component/s: fs/s3
• Target Version/s: 3.1.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 02/Nov/16 18:16
• Updated: 12/Jun/23 21:43
• Resolved: 22/Nov/17 17:28

Description

A goal of this code is "support O(1) commits to S3 repositories in the presence of failures". Implement it, including whatever is needed to demonstrate the correctness of the algorithm. (that is, assuming that s3guard provides a consistent view of the presence/absence of blobs, show that we can commit directly).

I consider ourselves free to expose the blobstore-ness of the s3 output streams (ie. not visible until the close()), if we need to use that to allow us to abort commit operations.

Issue Links

• breaks: HADOOP-15631 Remove transient dependency on hadoop-hdfs-client
• breaks: MAPREDUCE-7014 Fix java doc errors in jdk1.8
• contains: HADOOP-14714 handle InternalError in bulk object delete through retries
• contains: HADOOP-14717 Add StreamCapabilities support to s3a
• contains: HADOOP-14423 s3guard will set file length to -1 on a putObjectDirect(stream, -1) call
• contains: HADOOP-15228 S3A Retry policy to retry on NoResponseException
• contains: MAPREDUCE-6961 Pull up FileOutputCommitter.getOutputPath to PathOutputCommitter
• depends upon: HADOOP-13449 S3Guard: Implement DynamoDBMetadataStore.
• depends upon: MAPREDUCE-6823 FileOutputFormat to support configurable PathOutputCommitter factory
• depends upon: MAPREDUCE-6956 FileOutputCommitter to gain abstract superclass PathOutputCommitter
• duplicates: HADOOP-13205 S3A to support custom retry policies; failfast on unknown host
• duplicates: HADOOP-13811 s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class
• duplicates: HADOOP-14303 Review retry logic on all S3 SDK calls, implement where needed
• duplicates: HADOOP-14381 S3AUtils.translateException to map 503 reponse to => throttling failure
• duplicates: MAPREDUCE-6823 FileOutputFormat to support configurable PathOutputCommitter factory
• incorporates: HADOOP-13205 S3A to support custom retry policies; failfast on unknown host
• incorporates: HADOOP-13811 s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class
• incorporates: HADOOP-13967 S3ABlockOutputStream to support plugin point for different multipart strategies
• incorporates: HADOOP-13968 S3a FS to support "__magic" path for the special "unmaterialized" writes
• incorporates: HADOOP-13969 S3A to support commit(path) operation, which commits all pending put commits in a path
• incorporates: HADOOP-14303 Review retry logic on all S3 SDK calls, implement where needed
• incorporates: HADOOP-14381 S3AUtils.translateException to map 503 reponse to => throttling failure
• incorporates: MAPREDUCE-6823 FileOutputFormat to support configurable PathOutputCommitter factory
• incorporates: HADOOP-14859 Shaded AWS library stops s3a recognising ConnectTimeoutException
• is depended upon by: HADOOP-13761 S3Guard: implement retries for DDB failures and throttling; translate exceptions
• is depended upon by: HADOOP-14831 Über-jira: S3a phase IV: Hadoop 3.1 features
• is duplicated by: HADOOP-14971 Merge S3A committers into trunk
• is duplicated by: HADOOP-15003 Merge S3A committers into trunk: Yetus patch checker
• is duplicated by: HADOOP-13574 Unnecessary file existence check causes problems with S3
• is duplicated by: HADOOP-13912 S3a Multipart Committer (avoid rename)
• is duplicated by: HADOOP-15087 S3A to support writing directly to the destination dir without creating temp directory to avoid rename
• is related to: HDFS-13713 Add specification of Multipart Upload API to FS specification, with contract tests
• is related to: HADOOP-15079 ITestS3AFileOperationCost#testFakeDirectoryDeletion failing after OutputCommitter patch
• is related to: HADOOP-14303 Review retry logic on all S3 SDK calls, implement where needed
• is related to: HADOOP-14584 WASB to support high-performance commit protocol
• is related to: HADOOP-15890 Some S3A committer tests don't match ITest* pattern; don't run in maven
• is related to: HBASE-20431 Store commit transaction for filesystems that do not support an atomic rename
• is related to: SPARK-10063 Remove DirectParquetOutputCommitter
• is related to: SPARK-18883 FileNotFoundException on _temporary directory
• is related to: HADOOP-14161 Failed to rename file in S3A during FileOutputFormat commitTask
• is related to: HADOOP-13912 S3a Multipart Committer (avoid rename)
• is related to: HADOOP-18600 Hadoop 2.x should support s3a committers
• is related to: MAPREDUCE-6974 Add standard configuration keys for HTrace values, propagate across to MR committers if set
• is related to: MAPREDUCE-7060 Cherry Pick PathOutputCommitter class/factory to branch-3.0
• relates to: HADOOP-13846 S3A to implement rename(final Path src, final Path dst, final Rename... options)
• relates to: HIVE-16295 Add support for using Hadoop's S3A OutputCommitter
• relates to: SPARK-18512 FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A
• relates to: SPARK-22217 ParquetFileFormat to support arbitrary OutputCommitters
• supercedes: HADOOP-14161 Failed to rename file in S3A during FileOutputFormat commitTask

Sub-Tasks

• S3ABlockOutputStream to support plugin point for different multipart strategies: Resolved
• S3a FS to support "__magic" path for the special "unmaterialized" writes: Resolved
• S3A to support commit(path) operation, which commits all pending put commits in a path: Resolved
• Merge S3A committers into trunk: Resolved
• Merge S3A committers into trunk: Yetus patch checker: Resolved