[MAPREDUCE-6961] Pull up FileOutputCommitter.getOutputPath to PathOutputCommitter

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 3.0.0-beta1
• Component/s: mrv2
• Target Version/s: 3.1.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 19/Sep/17 09:46
• Updated: 22/Mar/18 03:19
• Resolved: 26/Feb/18 14:56

Description

SPARK-21549 has shown that downstream code is relying on the internal property

if we pulled FileOutputCommitter.getOutputPath to the PathOutputCommitter of MAPREDUCE-6956, then there'd be a public/stable way to get this. Admittedly, it does imply that the committer will always have some output path, but FileOutputFormat depends on that anyway.

Issue Links

• Is contained by: HADOOP-13786 Add S3A committers for zero-rename commits to S3 endpoints
• relates to: MAPREDUCE-6823 FileOutputFormat to support configurable PathOutputCommitter factory
• relates to: SPARK-21549 Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs
• relates to: MAPREDUCE-6956 FileOutputCommitter to gain abstract superclass PathOutputCommitter