[SPARK-44209] Expose amount of shuffle data available on the node

• Type: New Feature
• Status: Open
• Priority: Trivial
• Resolution: Unresolved
• Affects Version/s: 3.4.1
• Component/s: Shuffle
• Labels: pull-request-available

Language

Java

• Assignee: Unassigned
• Reporter: Deependra Patel
• Shepherd: Abhishek Modi
• Created: 27/Jun/23 11:41
• Updated: 27/Nov/23 00:19

Description

ShuffleMetrics doesn't have metrics like 
"totalShuffleDataBytes" and "numAppsWithShuffleData", these metrics are per node published by External Shuffle Service.
 
Adding these metrics would help in -
1. Deciding if we can decommission the node if no shuffle data present
2. Better live monitoring of customer's workload to see if there is skewed shuffle data present on the node

Issue Links

• links to: [Github] Pull Request #42071 (Deependra-Patel)
• links to: [Github] Pull Request #42071 (Deependra-Patel)
• links to: GitHub Pull Request #42071