[SPARK-32643] [Cleanup] Consolidate state kept in ExecutorDecommissionInfo with TaskSetManager.tidToExecutorKillTimeMapping

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: Spark Core
• Assignee: Devesh Agrawal
• Reporter: Devesh Agrawal
• Created: 18/Aug/20 00:29
• Updated: 26/Aug/20 22:18
• Resolved: 26/Aug/20 22:18

Description

The decommissioning state is a bit fragment across two places in the TaskSchedulerImpl:

• https://github.com/apache/spark/pull/29014/ stored the incoming decommission info messages in TaskSchedulerImpl.executorsPendingDecommission.

• While https://github.com/apache/spark/pull/28619/ was storing just the executor end time in the map TaskSetManager.tidToExecutorKillTimeMapping (which in turn is contained in TaskSchedulerImpl).

While the two states are not really overlapping, its a bit of a code hygiene concern to save this state in two places. 

With https://github.com/apache/spark/pull/29422, TaskSchedulerImpl is emerging as the place where all decommissioning book keeping is kept within the driver. So consolidate the information in tidToExecutorKillTimeMapping into ExecutorDecommissionInfo.

 

 

Issue Links

• links to: [Github] Pull Request #29452 (agrawaldevesh)
• links to: [Github] Pull Request #29452 (agrawaldevesh)