Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-17124

The PyFlink Job runs into infinite loop if the Python UDF imports job code

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      If the UDF file imports job code directly or indirectly, the PyFlink Job will run into a infinite loop as follows:

      • submit job
      • execute job
      • launch UDF worker
      • import UDF
      • (If the job file is depended by UDF or imported as the top level module) import job code
      • (If the job code is executed outside the "if _name_ == '_main_':") launch gateway server and submit job to local executor
      • execute job in local mode
      • launch UDF worker
      • import UDF
      • import job code
        ...
        This infinite loop will create new Java processes and Python processes endlessly until the resources on the machine are exhausted. We should fix it ASAP.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            hxbks2ks Huang Xingbo
            hxbks2ks Huang Xingbo
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m

                Slack

                  Issue deployment