Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3897

capacity scheduler - maxActiveApplicationsPerUser calculation can be wrong

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.2
    • Component/s: mrv2
    • Labels:
      None

      Description

      The capacity scheduler calculates the maxActiveApplications and the maxActiveApplicationsPerUser based on the config yarn.scheduler.capacity.maximum-applications or default 10000.

      MaxActiveApplications = max ( ceil ( clusterMemory/minAllocation * maxAMResource% * absoluteMaxCapacity), 1)

      MaxActiveAppsPerUser = max( ceil (maxActiveApplicationsComputedAbove * (userLimit%/100) * userLimitFactor), 1)

      maxActiveApplications is already multiplied by the queue absolute MAXIMUM capacity, so if max capacity > capacity and if you have user limit factor 1 (which is the default) and only 1 user is running, that user will not be allowed to use over the queue capacity, so having it relative to MAX capacity doesn't make sense. That user could easily end up in a deadlock and all its space used by application masters.

      1. MAPREDUCE-3897-2.txt
        5 kB
        Eric Payne
      2. MAPREDUCE-3897-1.txt
        3 kB
        Eric Payne
      3. MAPREDUCE-3897-1.txt
        3 kB
        Eric Payne

        Activity

        Thomas Graves created issue -
        Eric Payne made changes -
        Field Original Value New Value
        Assignee Eric Payne [ eepayne ]
        Eric Payne made changes -
        Attachment MAPREDUCE-3897-1.txt [ 12516616 ]
        Eric Payne made changes -
        Attachment MAPREDUCE-3897-1.txt [ 12516619 ]
        Eric Payne made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Eric Payne made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Eric Payne made changes -
        Attachment MAPREDUCE-3897-2.txt [ 12516874 ]
        Eric Payne made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.23.2 [ 12319851 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Eric Payne
            Reporter:
            Thomas Graves
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development