[SPARK-42736] Display executor remove reason statistics for clusters which don't have dynamic allocation

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.3.2
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Yeachan Park
• Shepherd: Holden Karau
• Created: 09/Mar/23 14:56
• Updated: 09/Mar/23 14:57

Description

The remove reason statistics do not seem to be enabled for clusters which do not have dynamic allocation enabled. For clusters which do have dynamic allocation, we see this:

ExecutorMonitor: Executor 2 is removed. Remove reason statistics: (gracefully decommissioned: 2, decommision unfinished: 0, driver killed: 0, unexpectedly exited: 0).

Something similar would be helpful for clusters which don't have dynamic allocation, as executors can still be killed for whatever reason and it would be helpful to see if those killed executors were able to gracefully decommission.