[BEAM-8617] Tear down the DoFns upon the control service termination in Python SDK harness

• Type: Improvement
• Status: Resolved
• Priority: P2
• Resolution: Duplicate
• Fix Version/s: 2.18.0
• Component/s: sdk-py-harness
• Assignee: sunjincheng
• Reporter: sunjincheng
• Created: 12/Nov/19 13:37
• Updated: 24/Jul/20 20:02
• Resolved: 25/Nov/19 06:00

Description

Per the discussion in the ML can be found [1], the teardown of DoFns should be supported in the portability framework. It happens at two places:
1) Upon the control service termination
2) Tear down the unused DoFns periodically

The aim of this JIRA is to add support to teardown the DoFns upon the control service termination for Python SDK harness.

[1] https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E

Issue Links

• duplicates: BEAM-8151 Allow the Python SDK to use many many threads
• Is contained by: BEAM-8325 SdkWorker should be stopped in SdkHarness
• links to: GitHub Pull Request #10073