[HIVE-27494] Deduplicate the task result that generated by more branches in union all

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-beta-1
• Labels: pull-request-available
• Assignee: Zhihua Deng
• Reporter: Zhihua Deng
• Created: 10/Jul/23 07:51
• Updated: 23/Apr/24 03:35
• Resolved: 31/Jul/23 08:25

Description

HIVE-23891 adds the ability to deduplicate the task result that under the directory,

<table-dir>/<staging-dir>/_tmp.-ext-10000/<dynamic-partition-dir>/HIVE_UNION_SUBDIR_1,

but turns out to ignore taking the same action to the output directory for the same query:

<table-dir>/<staging-dir>/_tmp.-ext-10000/<dynamic-partition-dir>/HIVE_UNION_SUBDIR_2.

So user may still have the same data duplication problem upon multiple tez task attempts.

Issue Links

• relates to: HIVE-23891 UNION ALL and multiple task attempts can cause file duplication
• links to: GitHub Pull Request #4479