Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-19976 FLIP-136: Improve interoperability between DataStream and Table API
  3. FLINK-23372

Disable AllVerticesInSameSlotSharingGroupByDefault in DataStream batch mode

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      In order to unify the behavior of DataStream API and Table API batch mode, we should disable AllVerticesInSameSlotSharingGroupByDefault also in DataStream API.

      FLINK-20001 reverted setting this flag but without concrete arguments and the following comment: reconsider actually setting this flag in the future

      After a offline chat with Zhu Zhu, we should introduce this again for consistency:

      The goal to assign different regions to different slot sharing groups by default is to reduce waste of resources. In batch jobs, there can be one region which has data dependency on another region. And the resource computation for slots and managed memory will be affected:
        1. If these regions are in the same slot sharing group, the group will require a large slot which can host tasks from both the regions.
        2. In managed memory fraction computing, tasks from both regions will be considered to compete for managed memory, so each task will be assigned with a smaller managed memory fraction (FLIP-53).
      
      However, those regions will not run at the same time and results in a waste of resources. 
      
      For streaming jobs, all tasks will run at the same time. So assigning them to the same slot sharing group will not result resource waste.
      

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              twalthr Timo Walther
              Reporter:
              twalthr Timo Walther

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment