[HIVE-26758] Allow use scratchdir for staging final job

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0.0-alpha-2
• Fix Version/s: 4.0.0-beta-1
• Component/s: Query Planning
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Yi Zhang Assign to me
• Reporter: Yi Zhang
• Created: 18/Nov/22 00:30
• Updated: 27/Sep/23 00:21
• Resolved: 06/Dec/22 22:01

Description

The query results are staged in stagingdir that is relative to the destination path <destination_dir>/<staging_dir>/

during blobstorage optimzation HIVE-17620 final job is set to use stagingdir.

HIVE-15215 mentioned the possibility of using scratch for staging when write to S3 but it was long time ago and no activity.

 

This is to allow final job to use hive.exec.scratchdir as the interim jobs, with a configuration 

hive.use.scratchdir.for.staging

This is useful for cross Filesystem, user can use local source filesystem instead of remote filesystem for the staging.

Issue Links

• links to: GitHub Pull Request #3831
• links to: GitHub Pull Request #4499
• links to: GitHub Pull Request #4500