[HADOOP-14584] WASB to support high-performance commit protocol

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 3.0.0-alpha3
• Component/s: fs/azure
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 23/Jun/17 18:14
• Updated: 13/Apr/18 11:32
• Resolved: 13/Apr/18 11:32

Description

Once MAPREDUCE-6823 allows FileOutputFormat to take alternate committers, and HADOOP-13786 provides the first implementation and tests of a blobstore specific committer, WASB could do its own. The same strategy: upload uncommitted blobs and coalesce at the end should work; the same marshalling of lists of etags probably work the same, though there will inevitably be some subtle differences.

Issue Links

• depends upon: MAPREDUCE-6823 FileOutputFormat to support configurable PathOutputCommitter factory
• relates to: HADOOP-13786 Add S3A committers for zero-rename commits to S3 endpoints