I submit Python Beam jobs to our Flink cluster with the PortableRunner through a remote job server. If a job finishes within a few seconds or minutes, the return status (including a dump of any Python exceptions in case there was an error) is returned to the client upon completion.
If the job, however, runs for longer (say) hours, then the client and job server seem to lose connection. This results in the client hanging forever until I press Ctrl+C to terminate it, even long after the actual job has completed (which has no effect whatsoever on the actual job).
Example pseudo job:
If the pipeline finishes quickly, it looks like this from the client's perspective:
If the job runs for longer, then the with statement never finishes and I have to abort the Python script with Ctrl+C: