[HADOOP-15107] Stabilize/tune S3A committers; review correctness & docs

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.2
• Component/s: fs/s3
• Target Version/s: 3.2.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 10/Dec/17 22:23
• Updated: 30/Oct/18 19:32
• Resolved: 30/Aug/18 14:24

Description

I'm writing about the paper on the committers, one which, being a proper paper, requires me to show the committers work.

1. define the requirements of a "Correct" committed job (this applies to the FileOutputCommitter too)
2. show that the Staging committer meets these requirements (most of this is implicit in that it uses the V1 FileOutputCommitter to marshall .pendingset lists from committed tasks to the final destination, where they are read and committed.
3. Show the magic committer also works.

Issue Links

• contains: HADOOP-15095 S3a committer factory to warn when default FileOutputFormat committer is created
• contains: HADOOP-15470 S3A staging committers to not log FNFEs on job abort listings
• is related to: HADOOP-15782 Clarify committers.md around v2 failure handling
• is related to: HADOOP-15890 Some S3A committer tests don't match ITest* pattern; don't run in maven