[YARN-6406] Remove SchedulerRequestKeys when no more pending ResourceRequest

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0, 2.7.3, 3.0.0-alpha2
• Fix Version/s: 2.9.0, 3.0.0-alpha4
• Hadoop Flags: Reviewed
• Assignee: Arun Suresh
• Reporter: Arun Suresh
• Created: 28/Mar/17 21:22
• Updated: 17/Apr/17 15:41
• Resolved: 16/Apr/17 19:29

Description

YARN-5540 introduced some optimizations to remove satisfied SchedulerKeys from the AppScheduleingInfo. It looks like after YARN-6040, ScedulerRequestKeys are removed only if the Application sends a 0 numContainers requests. While earlier, the outstanding schedulerKeys were also remove as soon as a container is allocated as well.

An additional optimization we were hoping to include is to remove the ResourceRequests itself once the numContainers == 0, since we see in our clusters that the RM heap space consumption increases drastically due to a large number of ResourceRequests with 0 num containers.