[HIVE-23140] Optimise file move in CTAS - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Minor
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 4.0.0-alpha-1
Component/s: HiveServer2
Labels:
None

Description

FileSinkOperator can be optimized to run file move operation (/_tmp.~~ext --> /-ext~~) in parallel fashion. Currently it invokes Utilities.moveSpecifiedFileStatus and renames in sequential mode causing delays in cloud storage. FS rename can be used (S3A internally has parallel rename operation).

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

HIVE-23140.1.patch
06/Apr/20 06:43
1 kB
Rajesh Balamohan

Activity

People

Assignee:: Rajesh Balamohan

Reporter:: Rajesh Balamohan

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 06/Apr/20 06:43

Updated:: 17/Nov/22 08:50

Resolved:: 07/Apr/20 05:32