[BEAM-8103] Stop job service when pipeline execution finishes

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-flink
• Labels: portability-flink
• Assignee: Unassigned
• Reporter: Kyle Weaver
• Created: 27/Aug/19 23:13
• Updated: 04/Jun/22 13:33

Description

Currently, job servers are shut down when the Python script exits [1]. A better long-term solution would be to shut them down instead when the pipeline is finished executing, such as [2]. This will put resource management in a common code path that is less error-prone.

 

[1] https://github.com/apache/beam/blob/c5f43342f914fc8ff367b86fb9294c38436ed3ce/sdks/python/apache_beam/runners/portability/job_server.py#L73

[2] https://github.com/apache/beam/blob/c5f43342f914fc8ff367b86fb9294c38436ed3ce/sdks/python/apache_beam/runners/portability/portable_runner.py#L451

Issue Links

• is related to: BEAM-7973 Python doesn't shut down Flink job server properly