Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-25410

Flink CLI should exit when app is accepted with detach mode on Yarn

Agile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Why

      In our internal streaming platform, we will use flink-cli tool to submit Flink streaming application on Yarn.

      However when encountering Hadoop cluster down and then lots of flink apps need to be resubmitted, the submitter of worker in our platform will hang at this time.

      Because the Yarn cluster resources are tight and the scheduling efficiency becomes low when lots of apps needs to be started.

      And flink-cli will not exit until the app status changes to running.

      In addition, I also think there is no need to wait when app status is accepted with detach mode on Yarn.

      How

      When app in accpeted status, flink-cli should exit directly to release submitter worker process resource. The PR could refer to : https://github.com/apache/flink/blob/f191becdb42d6df823a103dc4f787c4737baa8e7/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L1224

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zuston Junfan Zhang
            zuston Junfan Zhang

            Dates

              Created:
              Updated:

              Slack

                Issue deployment