[HIVE-25990] Optimise multiple copies in case of CTAS in external tables for Object stores

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Ayush Saxena
• Reporter: Ayush Saxena
• Created: 26/Feb/22 13:46
• Updated: 28/Jul/23 07:23
• Resolved: 14/Mar/22 04:48

Description

Presently for CTAS with external tables, there are two renames, operations, one from tmp to _ext and then from _ext to actual target.
In case of object stores, the renames lead to actual copy. Avoid renaming by avoiding rename from tmp to _ext, but by creating a list of files to be copied in that directly, which can be consumed in the move task, to copy directly from tmp to actual target.

Issue Links

• causes: HIVE-26382 Stats generation fails during CTAS for external partitioned table.
• causes: HIVE-27536 Merge task must be invoked after optimisation for external CTAS queries
• links to: Github PR #3058
• links to: GitHub Pull Request #3058