[SPARK-35627] Decommission executors in batches to avoid overloading network by block migrations.

• Type: Sub-task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: abhishek kumar tiwari
• Created: 03/Jun/21 10:18
• Updated: 03/Jun/21 10:22

Description

Currenlty, each executor is asked to starts offloading rdd and shuffle blocks as soon it is decommissioned. This can overload the network bandwidth of the application.

 

We should limit the number of executors migrating rdd and shuffle blocks at one time. This would be configurable to enable users to decommission executors conservatively or aggressively based on the use cases.

Issue Links

• links to: [Github] Pull Request #32766 (q2w)
• links to: [Github] Pull Request #32766 (q2w)