Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-21222

Support loopback mode to allow Python UDF worker and client reuse the same Python VM

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Instead of launching a separate Python process, it will reuse the Python process of the client side to execute Python UDFs if you are running jobs locally. This makes it easy for Python UDFs debugging.

    Description

      Currently the Python process for the client and the Python UDF worker runs in different Python processes. It would bring the following benefits if we could support loopback mode to allow Python UDF worker and client reuse the same Python VM:
      1) It could greatly reduce the job initialization overhead in mini-cluster mode as there is no need to start the Python process any more.
      2) It makes the debug easier.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            hxbks2ks Huang Xingbo
            dian.fu Dian Fu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment