[BEAM-14213] Add support for Batched DoFns in the Python SDK

• Type: Improvement
• Status: Open
• Priority: P2
• Resolution: Unresolved
• Component/s: sdk-py-core
• Assignee: Unassigned
• Reporter: Brian Hulette
• Created: 30/Mar/22 20:58
• Updated: 05/Jun/22 01:07

Description

Add an implementation for https://s.apache.org/batched-dofns to the Python SDK.

Issue Links

• blocks: BEAM-14044 Hook In Batching DoFn Apis to RunInference
• is related to: BEAM-14540 Native implementation for serialized Rows to/from Arrow
• links to: GitHub Pull Request #17253

Sub-Tasks

• Add API and construction-time validation for process_batch (with assumptions): Resolved
• Add @DoFn.yields_batches and @DoFn.yields_elements decorators to override defaults: Open
• MVP for SDK worker changes to support process_batch: Resolved
• More verbose errors when BatchConverter fails to match: Open
• Consider providing a dynamic API for declaring batch input type: Open
• Add a TimestampedBatch analogue for TimestampedValue: Open
• batch-consuming DoFns should estimate byte size: Open
• Support per-key DoFn params in process_batch: Open