Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6407

Improve and fix locks of RM scheduler

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.7.1
    • Component/s: fairscheduler
    • Labels:
      None
    • Environment:

      CentOS 7, 1 Gigabit Ethernet

    • Target Version/s:

      Description

      First´╝îthis issue dose not duplicate the YARN-3091.
      In our cluster, we have 5k nodes, and the server is configured with 1 Gigabit Ethernet. So network is bottleneck in our cluster.
      We must distcp data from warehouse, because of 1 Gigabit Ethernet, we must set yarn.scheduler.fair.max.assign to 5, or must lead to hotspot.
      The setting that max.assign is 5 lead to the assigned ability decreased. So we start the ContinuousSchedulingThread.
      As more applicaitons running in our cluster , and with ContinuousSchedulingThread, the problem of lock contention is more serious.
      In our cluster, the callqueue of ApplicationMasterSeriver's rpc is high occasionally. we worried that more problem occure in future with more application are running.
      Here is our logical graph:
      "1 Gigabit Ethernet" and "data hot spot" ==> "set yarn.scheduler.fair.max.assign to 5" ==> "ContinuousSchedulingThread is started" and "more applcations" => "lock contention"
      I know YARN-3091 solved this problem, but the patch aims that change the object lock to read write lock. This change is still Coarse-Grained. So I think we lock the resources or not lock the large section code.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              zhengchenyu zhengchenyu
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 2m
                2m
                Remaining:
                Remaining Estimate - 2m
                2m
                Logged:
                Time Spent - Not Specified
                Not Specified