[BEAM-11249] Read a reasonable amount of data per chunk.

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.26.0
• Component/s: sdk-py-core
• Assignee: Robert Bradshaw
• Reporter: Robert Bradshaw
• Created: 12/Nov/20 00:34
• Updated: 21/Dec/20 20:51
• Resolved: 21/Dec/20 20:49

Description

In read_csv (and others), we are reading only 100 rows or so at a time. This gives kb-sized chunks, the reader is likely optimal in the mb-sized ranges.

Issue Links

• links to: GitHub Pull Request #13312