[ARROW-14429] [C++] RecordBatchFileReader performance really bad in S3

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.0.0
• Fix Version/s: 7.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/29993
• Assignee: Weston Pace
• Reporter: Lingkai Kong
• Created: 21/Oct/21 22:18
• Updated: 11/Jan/23 08:39
• Resolved: 17/Jan/22 14:40

Description

We are using RecordBatchFileWriter to write Arrow type directly to S3 using the S3FileSystem, then using RecordBatchFileReader to read from S3. The write is pretty efficient, write a 50MB finishes within 0.2s. But reading that file is taking 30s, which is definitely too long. Then I did several tests:

1. I tried to use S3FileSystem to read the file into bytes, it's only taking 1s. which somehow makes me believe it's an issue with RecordBatchFileReader
2. Half the size (around 25MB), with RecordBatchFileReader took 17s, without RecordBatchFileReader took 0.28s
3. Double the size (around 100MB), with RecordBatchFileReader took 61s, without RecordBatchFileReader took 2.3s
4. I tried to get all bytes using S3FileSystem first, then create a reader from the bytes. Then read all context from the reader, it's only taking 0.1s. 

Issue Links

• duplicates: ARROW-14577 [C++] Enable fine grained IO for async IPC reader
• links to: GitHub Pull Request #11535