[HIVE-21214] MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Deepak Jaiswal
• Reporter: Deepak Jaiswal
• Created: 05/Feb/19 08:38
• Updated: 06/Feb/19 16:32
• Resolved: 06/Feb/19 16:32

Description

For a given task, if there is more than one attempt then deduplication logic kicks in.

Utilities.compareTempOrDuplicateFiles()

The logic uses file size and picks the one with largest size. This logic is very fragile.

ideally, it should pick the successful attempt's file.

However, a simpler solution is to pick the newest attempt and also checking the file size for the newest attempt is the largest.

If not, throw an exception.

 

cc gopalv thejas jdere ekoifman

Issue Links

• links to: RB