[ARROW-17033] [C++] Add GCS connection pool size option

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 8.0.0
• Component/s: C++
• Labels: GCP good-first-issue performance
• External issue URL: https://github.com/apache/arrow/issues/20314

Language

C++

• Assignee: Unassigned
• Reporter: Leonhard Gruenschloss
• Created: 10/Jul/22 12:07
• Updated: 11/Jan/23 11:48

Description

Multi-threaded read performance in Arrow's GCS file system implementation currently is relatively low. Given the high latency of cloud blob systems like GCS, a common strategy is to use many concurrent readers (if the system has enough memory to support that), e.g. using 100 threads.

The GCS client library offers a ConnectionPoolSize option. If this option is set to a value that's too low, concurrency is throttled. At the moment, this is not exposed in GcsOptions, consequently limiting multi-threaded throughput.

Instead of exposing this option, an alternative implementation strategy could be to use the same value as set by arrow::io::SetIOThreadPoolCapacity.