[IMPALA-10791] Add Support of Batch Reading for Spilling to Remote FS

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.2.0
• Component/s: Backend
• Epic Link: Spill to S3 Support
• Epic Color: ghx-label-14
• Assignee: Yida Wu
• Reporter: Yida Wu
• Created: 12/Jul/21 22:23
• Updated: 29/Nov/22 10:40
• Resolved: 30/Sep/22 22:58

Description

Impala allows spilling to a remote filesystem, like S3. The rate of uploading the spilled data to the remote filesystem is fast, but the speed of reading from the remote is slow because each time only one page is read from the remote filesystem while we do per upload per file.

 

The task aims to improve the reading performance of the spilling to a remote filesystem by using batch reading.

Issue Links

• is duplicated by: IMPALA-10508 Add metrics for reading from remote scratch paths