[YARN-5326] Support for recurring reservations in the YARN ReservationSystem

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Fix Version/s: 2.9.0, 3.0.0
• Component/s: resourcemanager
• Hadoop Flags: Reviewed
• Release Note: Add native support for recurring reservations (good till cancelled) to enable periodic allocations of the same resources.
• Assignee: Carlo Curino
• Reporter: Subramaniam Krishnan
• Created: 07/Jul/16 01:41
• Updated: 22/Mar/18 04:56
• Resolved: 22/Mar/18 04:55

Description

YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle time explicitly, i.e. users can now "reserve" capacity ahead of time which is predictably allocated to them. Most SLA jobs/workflows are recurring so they need the same resources periodically. With the current implementation, users will have to make individual reservations for each run. This is an umbrella JIRA to enhance the reservation system by adding native support for recurring reservations.

Issue Links

• is depended upon by: HADOOP-14840 Tool to estimate resource requirements of an application pipeline based on prior executions
• is related to: YARN-2572 Enhancements to the ReservationSytem/Planner
• is related to: YARN-5211 Supporting "priorities" in the ReservationSystem
• is related to: YARN-2573 Integrate ReservationSystem with the RM failover mechanism
• is related to: YARN-1051 YARN Admission Control/Planner: enhancing the resource allocation model with time.
• relates to: YARN-6363 Extending SLS: Synthetic Load Generator

Sub-Tasks

• Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem: Resolved
• Placement Agent enhancements required to support recurring reservations in ReservationSystem: Resolved
• SharingPolicy enhancements required to support recurring reservations in ReservationSystem: Resolved
• Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem: Resolved
• API changes required to support recurring reservations in the YARN ReservationSystem: Resolved
• Add REST API for supporting recurring reservations: Resolved
• Update LowCost agents logic to take advantage of YARN-4358: Resolved
• Fix the logger in TestAlignedPlanner and TestGreedyReservationAgent: Resolved
• Adding reservation metrics in QueueMetrics: Resolved
• Validate that the duration of the periodic reservation is less than the periodicity: Resolved