[HIVE-1655] Adding consistency check at jobClose() when committing dynamic partitions

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Hadoop Flags: Reviewed
• Assignee: Ning Zhang
• Reporter: Ning Zhang
• Created: 17/Sep/10 23:20
• Updated: 16/Dec/11 23:59
• Resolved: 22/Sep/10 05:22

Description

In case of dynamic partition insert, FileSinkOperator generated a directory for a new partition and the files in the directory is named with '_tmp*'. When a task succeed, the file is renamed to remove the "_tmp", which essentially implement the "commit" semantics. A lot of exceptions could happen (process got killed, machine dies etc.) could left the _tmp files exist in the DP directory. These _tmp files should be deleted ("rolled back") at successful jobClose(). After the deletion, we should also delete any empty directories.