Details
Critical Description
The file merge stage had two tasks, which should create two files, but there was three files created.
By tracing the log, we found that there were two task attempts(one of them was a speculation) finished in one second by such a coincidence. Although the later one received a kill signal from AM, the rename operation was already done at that time, which cause the data duplication.
The rename operation was done at AbstractFileMergeOperator.closeOp(), the __ final path name was determined by the task attempt id rather than the task id. In this case, the final path ended with '000000_0' and '000000_1' rather than '000000'. IMHO, by making the final path name ended with task id without task attempt id, one task can only generate at most one file, which could solve this issue. But I don't know the side effects for changing the final path name.
This issue also affects other operators related to file renaming like JoinOperator and FileSinkOperator.
