[YARN-4879] Enhance Allocate Protocol to Identify Requests Explicitly

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-beta1
• Component/s: applications, resourcemanager
• Hadoop Flags: Reviewed
• Assignee: Subramaniam Krishnan
• Reporter: Subramaniam Krishnan
• Created: 26/Mar/16 01:34
• Updated: 30/Aug/18 03:09
• Resolved: 04/Oct/17 01:25

Description

For legacy reasons, the current allocate protocol expects expanded requests which represent the cumulative request for any change in resource constraints. This is not only very difficult to comprehend but makes it impossible for the scheduler to associate container allocations to the original requests. This problem is amplified by the fact that the expansion is managed by the AMRMClient which makes it cumbersome for non-Java clients as they all have to replicate the non-trivial logic. In this JIRA, we are proposing enhancement to the Allocate Protocol to allow AMs to identify requests explicitly.

Issue Links

• blocks: YARN-5415 Add support for NodeLocal and RackLocal OPPORTUNISTIC requests
• is blocked by: YARN-5132 Exclude generated protobuf sources from YARN Javadoc build
• is depended upon by: YARN-314 Schedulers should allow resource requests of different sizes at the same priority and location
• is depended upon by: YARN-2915 Enable YARN RM scale out via federation using multiple RM's
• is depended upon by: REEF-1288 Add a request ID to resource requests / allocated Evaluators
• is depended upon by: YARN-4485 [Umbrella] Capture per-application and per-queue container allocation latency
• is related to: YARN-1042 add ability to specify affinity/anti-affinity in container requests
• is related to: YARN-3870 Providing raw container request information for fine scheduling
• is related to: YARN-110 AM releases too many containers due to the protocol
• is related to: YARN-5907 [Umbrella] [YARN-1042] add ability to specify affinity/anti-affinity in container requests
• is related to: YARN-4902 [Umbrella] Generalized and unified scheduling-strategies in YARN
• relates to: YARN-2877 Extend YARN to support distributed scheduling

Sub-Tasks

• AM-RM protocol changes for identifying resource-requests explicitly: Resolved
• Changes in scheduler to identify resource-requests explicitly by allocation-id: Resolved
• Changes in AMRMClient for identifying resource-requests explicitly: Resolved
• Replace use of Priority in the Scheduling infrastructure with an opaque ShedulerRequestKey: Resolved
• Consider including allocationRequestId in NMContainerStatus to allow recovery in case of RM failover: Resolved