[SPARK-33763] Add metrics for better tracking of dynamic allocation

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Spark Core
• Assignee: Attila Zsolt Piros
• Reporter: Holden Karau
• Created: 11/Dec/20 18:56
• Updated: 12/May/21 20:02
• Resolved: 18/Feb/21 01:19

Description

We should add metrics to track the following:

1- Graceful decommissions & DA scheduled deletes

2- Jobs resubmitted

3- Fetch failures

4- Unexpected (e.g. non-Spark triggered) executor removals.

Issue Links

• is related to: SPARK-35369 Document ExecutorAllocationManager metrics
• links to: [Github] Pull Request #31450 (attilapiros)