[HIVE-3489] Use OutputCommitter to get rid of requirement of client being alive for long running queries

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Ashutosh Chauhan
• Created: 19/Sep/12 19:03
• Updated: 26/Aug/13 20:56

Description

There is a bunch of work (depending on query) which hive client does once query finishes. This has multiple effects which would be nice to avoid. Few of these work items include:

• Moving temp files / dirs around to their final destination.
• Creating empty files if required.
• Updating metastore with metadata.
• More ?

If we start using OutputCommitter then clients can fire and forget, they won't need to be around for all the time query is executing. Plus, since processing is going to happen in cluster, this further makes client lightweight.

Issue Links

• relates to: HIVE-4331 Integrated StorageHandler for Hive and HCat using the HiveStorageHandler