Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6406

Remove SchedulerRequestKeys when no more pending ResourceRequest

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 2.7.3, 3.0.0-alpha2
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      YARN-5540 introduced some optimizations to remove satisfied SchedulerKeys from the AppScheduleingInfo. It looks like after YARN-6040, ScedulerRequestKeys are removed only if the Application sends a 0 numContainers requests. While earlier, the outstanding schedulerKeys were also remove as soon as a container is allocated as well.

      An additional optimization we were hoping to include is to remove the ResourceRequests itself once the numContainers == 0, since we see in our clusters that the RM heap space consumption increases drastically due to a large number of ResourceRequests with 0 num containers.

        Attachments

        1. YARN-6406-branch-2.003.patch
          30 kB
          Arun Suresh
        2. YARN-6406-branch-2.002.patch
          30 kB
          Arun Suresh
        3. YARN-6406-branch-2.001.patch
          29 kB
          Arun Suresh
        4. YARN-6406.002.patch
          28 kB
          Arun Suresh
        5. YARN-6406.001.patch
          22 kB
          Arun Suresh

          Activity

            People

            • Assignee:
              asuresh Arun Suresh
              Reporter:
              asuresh Arun Suresh
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: