Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3446

FairScheduler headroom calculation should exclude nodes in the blacklist

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • fairscheduler
    • None
    • Reviewed

    Description

      FairScheduler HeadRoom calculation should exclude nodes in the blacklist.
      MRAppMaster does not preempt the reducers because for Reducer preemption calculation, headRoom is considering blacklisted nodes. This makes jobs to hang forever(ResourceManager does not assign any new containers on blacklisted nodes but availableResource AM get from RM includes blacklisted nodes available resource).
      This issue is similar as YARN-1680 which is for Capacity Scheduler.

      Attachments

        1. YARN-3446.000.patch
          11 kB
          Zhihai Xu
        2. YARN-3446.001.patch
          10 kB
          Zhihai Xu
        3. YARN-3446.002.patch
          11 kB
          Zhihai Xu
        4. YARN-3446.003.patch
          17 kB
          Zhihai Xu
        5. YARN-3446.004.patch
          16 kB
          Zhihai Xu
        6. YARN-3446.005.patch
          16 kB
          Zhihai Xu
        7. YARN-3446-branch-2.8.txt
          18 kB
          Vinod Kumar Vavilapalli

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6302 Preempt reducers after a configurable timeout irrespective of headroom

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              zxu Zhihai Xu
              zxu Zhihai Xu
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: