[HIVE-1492] FileSinkOperator should remove duplicated files from the same task based on file sizes

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Component/s: Query Processor
• Assignee: Ning Zhang
• Reporter: Ning Zhang
• Created: 29/Jul/10 00:48
• Updated: 17/Dec/11 00:03
• Resolved: 16/Aug/10 23:03

Description

FileSinkOperator.jobClose() calls Utilities.removeTempOrDuplicateFiles() to retain only one file for each task. A task could produce multiple files due to failed attempts or speculative runs. The largest file should be retained rather than the first file for each task.