[HIVE-21214] MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Deepak Jaiswal Assign to me
• Reporter: Deepak Jaiswal
• Created: 05/Feb/19 08:38
• Updated: 06/Feb/19 16:32
• Resolved: 06/Feb/19 16:32

Description

For a given task, if there is more than one attempt then deduplication logic kicks in.

Utilities.compareTempOrDuplicateFiles()

The logic uses file size and picks the one with largest size. This logic is very fragile.

ideally, it should pick the successful attempt's file.

However, a simpler solution is to pick the newest attempt and also checking the file size for the newest attempt is the largest.

If not, throw an exception.

 

cc Gopal Vijayaraghavan Thejas Nair Jason Dere Eugene Koifman

Issue Links

• links to: RB