[HIVE-587] Duplicate result from multiple TIPs of the same task

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.3.0
• Fix Version/s: 0.4.0
• Hadoop Flags: Reviewed
• Assignee: Zheng Shao
• Reporter: Zheng Shao
• Created: 27/Jun/09 01:38
• Updated: 17/Dec/11 00:07
• Resolved: 29/Jun/09 20:43

Description

On our cluster we found a job committed with duplicate output from different TIPs of the same Task (from FileSinkOperator).

The reason is that FileSinkOperator.commit can be called at multiple TIPs of the same task.

FileSinkOperator.jobClose() (which is called at the Hive Client side) should do either:
A. Get all successful TIPs and only move the output files of those TIPs to the output directory
B. Ignore TIPs from the JobInProgress, but only move one file out of potentially several output files

B is preferred because A might be slow (if the job finished and immediately got moved out of the JobTracker memory). Since we control the file name by ourselves, we know exactly what the file names are.