Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5650

Flink-python tests executing cost too long time

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 1.3.0
    • Fix Version/s: 1.3.0, 1.2.1
    • Component/s: Python API, Tests
    • Labels:

      Description

      When execute `mvn clean test` in flink-python, it will wait more than half hour after the console output below:
      -------------------------------------------------------
      T E S T S
      -------------------------------------------------------
      Running org.apache.flink.python.api.PythonPlanBinderTest
      log4j:WARN No appenders could be found for logger (org.apache.flink.python.api.PythonPlanBinderTest).
      log4j:WARN Please initialize the log4j system properly.
      log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

      The stack below:
      "main" prio=5 tid=0x00007f8d7780b800 nid=0x1c03 waiting on condition [0x0000700009fd8000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
      at java.lang.Thread.sleep(Native Method)
      at org.apache.flink.python.api.streaming.plan.PythonPlanStreamer.startPython(PythonPlanStreamer.java:70)
      at org.apache.flink.python.api.streaming.plan.PythonPlanStreamer.open(PythonPlanStreamer.java:50)
      at org.apache.flink.python.api.PythonPlanBinder.startPython(PythonPlanBinder.java:211)
      at org.apache.flink.python.api.PythonPlanBinder.runPlan(PythonPlanBinder.java:141)
      at org.apache.flink.python.api.PythonPlanBinder.main(PythonPlanBinder.java:114)
      at org.apache.flink.python.api.PythonPlanBinderTest.testProgram(PythonPlanBinderTest.java:83)
      at org.apache.flink.test.util.JavaProgramTestBase.testJobWithoutObjectReuse(JavaProgramTestBase.java:174)

      this is the jstack:
      https://gist.github.com/shijinkui/af47e8bc6c9f748336bf52efd3df94b0

        Issue Links

          Activity

          Hide
          shijinkui shijinkui added a comment -

          Now I have to excute `mvn clean test -pl '!flink-libraries/flink-python'` to exclude flink-python module.

          Show
          shijinkui shijinkui added a comment - Now I have to excute `mvn clean test -pl '!flink-libraries/flink-python'` to exclude flink-python module.
          Hide
          shijinkui shijinkui added a comment - - edited

          Also we can make PythonPlanStreamer support async executing in the thread pool, instead of blocked single process. Stephan Ewen

          I think such bad design unit test shouldn't be merge into the code base.

          Show
          shijinkui shijinkui added a comment - - edited Also we can make PythonPlanStreamer support async executing in the thread pool, instead of blocked single process. Stephan Ewen I think such bad design unit test shouldn't be merge into the code base.
          Hide
          Zentol Chesnay Schepler added a comment -

          This got very little to do with test design; the python tests are actually integration tests that simply run jobs like they would normally be run. The connection must be established synchronously as otherwise you cannot fulfill the purpose of open(); instantiate all resources that are required to do the data processing.

          We can however add a timeout to prevent it from being active for too long.

          Nevertheless i would like to know why the python process doesn't connect, since it does start up properly.

          Show
          Zentol Chesnay Schepler added a comment - This got very little to do with test design; the python tests are actually integration tests that simply run jobs like they would normally be run. The connection must be established synchronously as otherwise you cannot fulfill the purpose of open(); instantiate all resources that are required to do the data processing. We can however add a timeout to prevent it from being active for too long. Nevertheless i would like to know why the python process doesn't connect, since it does start up properly.
          Hide
          Zentol Chesnay Schepler added a comment -

          shijinkui Could you check out this branch, run the tests in flink-python and report back? https://github.com/zentol/flink/tree/5650_python_test_debug

          Show
          Zentol Chesnay Schepler added a comment - shijinkui Could you check out this branch, run the tests in flink-python and report back? https://github.com/zentol/flink/tree/5650_python_test_debug
          Hide
          shijinkui shijinkui added a comment -

          Chesnay Schepler, Your PR can work. flink-python UT cost one min. Very good

          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 01:09 min
          [INFO] Finished at: 2017-03-16T13:57:12+08:00
          [INFO] Final Memory: 22M/268M

          Show
          shijinkui shijinkui added a comment - Chesnay Schepler , Your PR can work. flink-python UT cost one min. Very good [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:09 min [INFO] Finished at: 2017-03-16T13:57:12+08:00 [INFO] Final Memory: 22M/268M
          Hide
          Zentol Chesnay Schepler added a comment -

          1.2: edd1065c73f3600929b1f731bc0b66255a56d26d
          1.3: c2bb5de6faeb314bd8cdb8c279a5cd9bfb4df059

          Show
          Zentol Chesnay Schepler added a comment - 1.2: edd1065c73f3600929b1f731bc0b66255a56d26d 1.3: c2bb5de6faeb314bd8cdb8c279a5cd9bfb4df059
          Hide
          shijinkui shijinkui added a comment -

          Good job, Thanks.

          Show
          shijinkui shijinkui added a comment - Good job, Thanks.
          Hide
          till.rohrmann Till Rohrmann added a comment -

          Chesnay Schepler, I think the encouraged way is to open a PR instead of linking private branches which are no longer accessible.

          Show
          till.rohrmann Till Rohrmann added a comment - Chesnay Schepler , I think the encouraged way is to open a PR instead of linking private branches which are no longer accessible.

            People

            • Assignee:
              Zentol Chesnay Schepler
              Reporter:
              shijinkui shijinkui
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development