Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-25318 Improvement of scheduler and execution for Flink OLAP
  3. FLINK-25334

Manage and share gateways of taskmanagers between jobs in session cluster

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.14.0, 1.12.4, 1.13.3
    • None
    • Runtime / Coordination
    • None

    Description

      Flink session cluster launches a certain number of taskmanagers and they won't exit during the cluster running. When a job is submitted to the cluster, taskmanagers will offer slots to `JobMaster` with location info. Then `JobMaster` will get `InetAddress` from the location and connect the taskmanager. This cause some performance loss and it isn't necessary in session cluster.

      It's better to manager the gateways of taskmanagers in JM and share them between jobs. When a taskmanager's gateway is exist and valid, the `JobMaster` can use it directly instead of creating a new one.

      Attachments

        Activity

          People

            Unassigned Unassigned
            zjureel Fang Yong
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated: