Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5846

Improve the fairscheduler attemptScheduler

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Duplicate
    • 2.7.1
    • 2.7.1
    • fairscheduler
    • CentOS-7.1

    • please pay attention to YARN-5139
    • Patch

    Description

      when I assign a container, we must consider two factor:
      (1) sort the queue and application, and select the proper request.
      (2) then we assure this request's host is just this node (data locality). or skip this loop!
      this algorithm regard the sorting queue and application as primary factor. when yarn consider data locality, for example, yarn.scheduler.fair.locality.threshold.node=1, yarn.scheduler.fair.locality.threshold.rack=1 (or yarn.scheduler.fair.locality-delay-rack-ms and yarn.scheduler.fair.locality-delay-node-ms is very large) and lots of applications are runnig, the process of assigning contianer becomes very slow.
      I think data locality is more important then the sequence of the queue and applications.
      I wanna a new algorithm like this:
      (1) when resourcemanager accept a new request, notice the RMNodeImpl, and then record this association between RMNode and request
      (2) when assign containers for node, we assign container by RMNodeImpl's association between RMNode and request directly
      (3) then I consider the priority of queue and applation. In one object of RMNodeImpl, we sort the request of association.
      (4) and I think the sorting of current algorithm is consuming, in especial, losts of applications are running, lots of sorting are called. so I think we should sort the queue and applicaiton in a daemon thread, because less error of queues's sequences is allowed.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            zhengchenyu Chenyu Zheng
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 1m
              1m
              Remaining:
              Remaining Estimate - 1m
              1m
              Logged:
              Time Spent - Not Specified
              Not Specified

              Slack

                Issue deployment