[BEAM-2643] Add TextIO and AvroIO read transforms that can read a PCollection of files

• Type: New Feature
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: sdk-py-core
• Assignee: Chamikara Madhusanka Jayalath
• Reporter: Chamikara Madhusanka Jayalath
• Created: 19/Jul/17 18:42
• Updated: 16/May/20 13:40
• Resolved: 10/Aug/17 20:40

Description

Java SDK now has TextIO.read_all() API that allows reading a massive number of files by moving from using the BoundedSource API (which may perform expensive source operations on the control plane) to using ParDo operations.

https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L170

This API should be added for Python SDK as well.

This form of reading files does not support dynamic work rebalancing for now. But this should not matter much when reading a massive number of relatively small files. In the future this API can support dynamic work rebalancing through Splittable DoFn.

cc: jkff

Issue Links

• links to: GitHub Pull Request #3661