[HIVE-16295] Add support for using Hadoop's S3A OutputCommitter

• Type: Sub-task
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Target Version/s: 3.1.0
• Assignee: Unassigned
• Reporter: Sahil Takiar
• Created: 24/Mar/17 20:20
• Updated: 03/Jan/20 17:11

Description

Hive doesn't have integration with Hadoop's OutputCommitter, it uses a NullOutputCommitter and uses its own commit logic spread across FileSinkOperator, MoveTask, and Hive.

The Hadoop community is building an OutputCommitter that integrates with S3Guard and does a safe, coordinate commit of data on S3 inside individual tasks (HADOOP-13786). If Hive can integrate with this new OutputCommitter there would be a lot of benefits to Hive-on-S3:

• Data is only written once; directly committing data at a task level means no renames are necessary
• The commit is done safely, in a coordinated manner; duplicate tasks (from task retries or speculative execution) should not step on each other

Issue Links

• is blocked by: HIVE-19217 Upgrade to Hadoop 3.1.0
• is blocked by: HIVE-18319 Upgrade to Hadoop 3.0.0
• is related to: HADOOP-15421 Stabilise/formalise the JSON _SUCCESS format used in the S3A committers
• is related to: HADOOP-13786 Add S3A committers for zero-rename commits to S3 endpoints
• relates to: HIVE-19321 Dynamic Partitioning Integration with Hadoop's S3A OutputCommitter
• links to: RB