[YARN-2745] Extend YARN to support multi-resource packing of tasks

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: nodemanager, resourcemanager, scheduler
• Assignee: Robert Grandl
• Reporter: Robert Grandl
• Created: 25/Oct/14 23:29
• Updated: 20/Sep/18 18:12

Description

In this umbrella JIRA we propose an extension to existing scheduling techniques, which accounts for all resources used by a task (CPU, memory, disk, network) and it is able to achieve three competing objectives: fairness, improve cluster utilization and reduces average job completion time.

Issue Links

• depends upon: YARN-3332 [Umbrella] Unified Resource Statistics Collection per node
• is related to: YARN-3332 [Umbrella] Unified Resource Statistics Collection per node
• requires: YARN-4056 Bundling: Searching for multiple containers in a single pass over {queues, applications, priorities}
• requires: HADOOP-12210 Collect network usage on the node
• requires: HADOOP-12211 Collect disks usages on the node

Sub-Tasks

• Enhance Node Managers to monitor and report the resource usage on machines: Patch Available
• Extend ask request to include additional fields: Open
• New task matching logic at the RM: Open
• Encapsulate additional group information in the AM to RM heartbeat: Open