[FLINK-33294] PyFlink minicluster is shut down after every job completion

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Test Infrastructure
• Assignee: Unassigned
• Reporter: Mehmet Aktas
• Created: 17/Oct/23 14:46
• Updated: 18/Oct/23 06:45

Description

What

A new PyFlink minicluster is created for each job. This significantly slows down execution.

We tried to configure the minicluster to prevent it from shutting down after every job completion, as shown here. This, however, did not seem to make any change.

Why

Our project relies on testing many PyFlink jobs on Github Actions. This issue is causing our testing process to take an unfeasibly long time.

How to reproduce

• Clone https://github.com/mfatihaktas/flink-playground
• make env; make install (dependencies: direnv, make)
• Run run_tests.sh
• Find the directory in which PyFlink writes the logs

• python -c "import pyflink;import os;print(os.path.dirname(os.path.abspath(pyflink.__file__))+'/log')" 

• Search in `flink-***.log` for “minicluster shutting down/created”. An example log is attached here for convenience: pyflink.log