[ARROW-9983] [C++][Dataset][Python] Use larger default batch size than 32K for Datasets API

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: C++
• Labels: dataset
• External issue URL: https://github.com/apache/arrow/issues/26008
• Assignee: Ben Kietzman
• Reporter: Wes McKinney
• Created: 12/Sep/20 20:01
• Updated: 11/Jan/23 08:10
• Resolved: 28/Sep/20 17:14

Description

Dremio uses 64K batch sizes. We could probably get away with even larger batch sizes (e.g. 256K or 1M) and allow memory-constrained users to elect a smaller batch size.

See example of some performance issues related to this in ARROW-9924

Issue Links

• is fixed by: ARROW-9924 [Python] Performance regression reading individual Parquet files using Dataset interface
• relates to: ARROW-9924 [Python] Performance regression reading individual Parquet files using Dataset interface