[YARN-4056] Bundling: Searching for multiple containers in a single pass over {queues, applications, priorities}

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: capacityscheduler, resourcemanager, scheduler
• Assignee: Robert Grandl
• Reporter: Srikanth Kandula
• Created: 16/Aug/15 23:09
• Updated: 27/Aug/15 18:44

Description

More than one container is allocated on many NM heartbeats. Yet, the current scheduler allocates exactly one container per iteration over queues, applications, priorities. When there are many queues, applications, or priorities allocating only one container per iteration can needlessly increase the duration of the NM heartbeat.

In this JIRA, we propose bundling. That is, allow arbitrarily many containers to be allocated in a single iteration over queues, applications and priorities.

Issue Links

• is required by: YARN-2745 Extend YARN to support multi-resource packing of tasks
• relates to: YARN-2967 New task matching logic at the RM

Sub-Tasks

• Expose Bundler logic and API as general purpose utility: Open
• Changes to AbstractYarnScheduler to incorporate the Bundler: Open
• Changes to Fair Scheduler to incorporate Bundler: Open
• Changes to Capacity Scheduler to incorporate Bundler: Open