[ARROW-12760] [C++][Python][R] S3FileSystem: IO thread parallelism limited to 8 threads

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 5.0.0
• Component/s: C++, Python, R
• Labels: filesystem performance pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/28502
• Assignee: David Li
• Reporter: Juan Galvez
• Created: 12/May/21 19:01
• Updated: 11/Jan/23 08:28
• Resolved: 26/May/21 14:51

Description

On Arrow 4, I'm not able to get more than 8 thread parallelism (for example when N threads call fs.ls() with different paths) for S3FileSystem created with pyarrow, because the default IO context is created with 8 thread limit (hardcoded), and I don't see a way to change this or pass a custom IO context through pyarrow APIs. I believe this affects other Arrow filesystem implementations as well.

Issue Links

• links to: GitHub Pull Request #10316