[SPARK-38447] Using Available Resources in Yarn Cluster Information in Spark Dynamic Allocation

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.1
• Component/s: YARN
• Assignee: Unassigned
• Reporter: Abhishek Dixit
• Created: 08/Mar/22 12:14
• Updated: 08/Mar/22 12:14

Description

Yarn Cluster Manager Provides information on available resources (VCores / Memory) in the Cluster via AM-RM heartbeat. In autoscaling, usually, latency to start executor containers on nodes that are immediately available in the cluster is comparatively lesser as compared to latency in adding new nodes to the cluster. Dynamic Allocation can leverage this information of the number of resources immediately available and latency in adding new nodes in deciding the number of executors to be requested from Yarn RM.

This improvement can be built in two parts:

1.  Infra to Send Available VCores and Memory information from Yarn AM-RM heartbeat response to ExecutorAllocationClient.
2. Leveraging Available VCores and Memory information in ExecutorAllocationManger to decide the number of executors to be requested from Yarn RM.

I'll create PRs for both the tasks one by one.

Sub-Tasks

• Sending Available Resources in Yarn Cluster Information to Spark Driver: In Progress