Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-45250

Support stage level task resource profile for yarn cluster when dynamic allocation disabled

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsDelete
    XMLWordPrintableJSON

Details

    Description

      https://github.com/apache/spark/pull/37268 has introduced a new feature that supports stage-level schedule task resource profile for standalone cluster when dynamic allocation is disabled. It's really cool feature, especially for ML/DL cases, more details can be found in that PR.

       

      The problem here is that the feature is only available for standalone cluster for now, but most users would also expect it can be used for other spark clusters like yarn and k8s.

       

      So I file this issue to track this task.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            wbo4958 Bobby Wang Assign to me
            wbo4958 Bobby Wang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment