[BEAM-7944] Improvements of portability framework to make it usable in other projects

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-core, sdk-py-harness
• Labels: portability
• Assignee: Unassigned
• Reporter: sunjincheng
• Created: 12/Aug/19 08:30
• Updated: 04/Jun/22 12:43

Description

The Flink community will use Beam's portability framework for its multi-language support, such as Python UDF execution. This is an umbrella JIRA which tracks all the improvement requirements collected from Flink community.

For details of the discussion can be found in [1].

[1] https://lists.apache.org/thread.html/4025a19ba267f0b1b842af20b63ac794306259a3226e07c87bda62ba@%3Cdev.beam.apache.org%3E

Issue Links

• contains: BEAM-7000 Consider adding support for batching state requests
• contains: BEAM-5468 Allow runner to set worker log level in Python SDK harness.
• contains: BEAM-7886 Make row coder a standard coder and implement in python
• contains: BEAM-5428 Implement cross-bundle state caching.

Sub-Tasks

• Allow runner to configure "semi_persist_dir" which is used in the SDK harness: Resolved
• Improves the interfaces of classes such as FnDataService, BundleProcessor, ActiveBundle, etc to change the parameter type from WindowedValue<T> to T: Resolved
• Add time-based cache threshold support in the Java data service: Resolved
• Add time-based cache threshold support in the data service of the Python SDK harness: Triage Needed
• Remove the Python 3 warning as it has already been supported: Resolved
• Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder: Resolved
• Make the input queue of the input buffer in Python SDK Harness size limited.: Resolved
• SdkWorker should be stopped in SdkHarness: Open
• Unify bundle register in Python SDK harness: Resolved
• Clean up useless null check.: Resolved