[HIVE-24936] Fix file name parsing and copy file move.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: HiveServer2
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Assignee: Harish JP
• Reporter: Harish JP
• Created: 25/Mar/21 05:40
• Updated: 17/Nov/22 08:47
• Resolved: 13/Sep/21 18:19

Description

The taskId and taskAttemptId is not extracted correctly for copy files (00001_02_copy_3) and when doing a move file of an incompatible copy file the rename utility generates wrong file names. Ex: 00001_02_copy_3 is renamed to 00001_02_copy_3_1 if 00001_02_copy_3 already exists, ideally it should be 00001_02_copy_N.

 

Incompatible files should be always renamed using the current task or it can get deleted if the file name conflicts with another task output file. Ex: if the input file name for a task is 00005_01 and is incompatible then if we move this file, it will be treated as an output file for task id 5, attempt 1 which if exists will try to generate the same file and fail and another attempt will be made. There will be 2 files 00005_01, 00005_02, the deduping code will remove 00005_01 resulting in data loss. There are other scenarios where the same can happen.

Issue Links

• fixes: HIVE-25130 alter table concat gives NullPointerException, when data is inserted from Spark
• links to: GitHub Pull Request #2120
• links to: GitHub Pull Request #2628