[BEAM-7514] Support streaming on the Python fn_api_runner

• Type: Bug
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: sdk-py-core
• Assignee: Unassigned
• Reporter: Pablo Estrada
• Created: 08/Jun/19 00:03
• Updated: 18/Jan/22 17:43

Description

A useful reference is the trigger manager for the Java direct runner: https://github.com/apache/beam/pull/3988/files

Sub-Tasks

• Support TestStream on the fn_api_runner: Open
• Add a watermark manager for the fn_api_runner: Resolved
• Make FnApiRunner work by executing ready elements instead of stages: Open
• Delete python DirectRunner after FnApiRunner fully supports Batch & Streaming: Open
• Refactor FnApiRunner into its own package: Resolved
• Add context managers for FnApiRunner to manage execution of each bundle: Resolved
• StateServicer should be owned by FnApiRunnerContextManager: Open
• Abstract bundle execution logic from stage execution logic: Resolved
• Track PCollection watermark across bundle executions: Resolved
• Split timers to multiple workers in ParallelBundleManager: Open
• Track performance of FnApiRunner over the changes to support streaming: Resolved
• A trigger manager for the FnApiRunner: Resolved
• Profile and improve performance for the local FnApiRunner: Open
• Support custom window fns on TriggerManager: Open
• Timers with same family ids in same stage (but different transforms) are buffered together: Open