[SPARK-45084] ProgressReport should include an accurate effective shuffle partition number

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.2
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: Siying Dong
• Reporter: Siying Dong
• Created: 05/Sep/23 23:17
• Updated: 15/Sep/23 01:37
• Resolved: 15/Sep/23 01:37

Description

Currently, there is a numShufflePartitions "metric" reported in 
StateOperatorProgress part of the progress report. However, the number is reported by aggregating executors so in the case of task retry or speculative executor, the metric is higher than number of shuffle partitions for the query plan. Number of shuffle partitions can be useful for reporting purpose so having a metric is helpful.

Issue Links

• links to: [Github] Pull Request #42822 (siying)
• links to: GitHub Pull Request #42822