[HIVE-21279] Avoid moving/rename operation in FileSink op for SELECT queries

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Query Planning
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Assignee: Vineet Garg
• Reporter: Vineet Garg
• Created: 15/Feb/19 23:34
• Updated: 22/Sep/24 11:41
• Resolved: 05/Mar/19 01:13

Description

Currently at the end of a job FileSink operator moves/rename temp directory to another directory from which FetchTask fetches result. This is done to avoid fetching potential partial/invalid files by failed/runway tasks. This operation is expensive for cloud storage. It could be avoided if FetchTask is passed on set of files to read from instead of whole directory.

Issue Links

• causes: HIVE-28530 Set files in thread safe manner in HiveSequenceFileInputFormat
• Dependent: HIVE-21386 Extend the fetch task enhancement done in HIVE-21279 to make it work with query result cache
• links to: GitHub Pull Request #582