[MAPREDUCE-6823] FileOutputFormat to support configurable PathOutputCommitter factory

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.0.0-alpha2
• Component/s: mrv2

Environment

Targeting S3 as the output of work

• Target Version/s: 3.1.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 14/Dec/16 16:25
• Updated: 22/Mar/18 06:07
• Resolved: 22/Mar/18 06:07

Description

In HADOOP-13786 I'm adding a custom subclass for FileOutputFormat, one which can talk direct to the S3A Filesystem for more efficient operations, better failure modes, and, most critically, as part of HADOOP-13345, atomic commit of output. The normal committer relies on directory rename() being atomic for this; for S3 we don't have that luxury.

To support a custom committer, we need to be able to tell FileOutputFormat (and implicitly, all subclasses which don't have their own custom committer), to use our new S3AOutputCommitter.

I propose:

1. FileOutputFormat takes a factory to create committers.
2. The factory to take a URI and TaskAttemptContext and return a committer
3. the default implementation always returns a FileOutputCommitter
4. A configuration option allows a new factory to be named
5. An S3AOutputCommitterFactory to return a FileOutputCommitter or new S3AOutputCommitter depending upon the URI of the destination.

Note that MRv1 already supports configurable committers; this is only the V2 API

Issue Links

• depends upon: MAPREDUCE-6956 FileOutputCommitter to gain abstract superclass PathOutputCommitter
• is depended upon by: HADOOP-14584 WASB to support high-performance commit protocol
• is depended upon by: HADOOP-13786 Add S3A committers for zero-rename commits to S3 endpoints
• is duplicated by: HADOOP-13786 Add S3A committers for zero-rename commits to S3 endpoints
• is part of: HADOOP-13786 Add S3A committers for zero-rename commits to S3 endpoints
• is related to: MAPREDUCE-6961 Pull up FileOutputCommitter.getOutputPath to PathOutputCommitter