Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-33294

PyFlink minicluster is shut down after every job completion

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Test Infrastructure
    • None

    Description

      What

      A new PyFlink minicluster is created for each job. This significantly slows down execution.

      We tried to configure the minicluster to prevent it from shutting down after every job completion, as shown here. This, however, did not seem to make any change.

      Why

      Our project relies on testing many PyFlink jobs on Github Actions. This issue is causing our testing process to take an unfeasibly long time.

      How to reproduce

      • Clone https://github.com/mfatihaktas/flink-playground
      • make env; make install (dependencies: direnv, make)
      • Run run_tests.sh
      • Find the directory in which PyFlink writes the logs
      • python -c "import pyflink;import os;print(os.path.dirname(os.path.abspath(pyflink.__file__))+'/log')" 
      • Search in `flink-***.log` for “minicluster shutting down/created”. An example log is attached here for convenience: pyflink.log 

      Attachments

        1. pyflink.log
          73 kB
          Mehmet Aktas

        Activity

          People

            Unassigned Unassigned
            mfatihaktas Mehmet Aktas
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: