[SPARK-33132] The 'Shuffle Read Size / Records' field in Stage Summary metrics was shown as 'NaN Undefined'

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.1.0
• Component/s: Spark Core
• Assignee: huangkunlun
• Reporter: huangkunlun
• Created: 13/Oct/20 11:36
• Updated: 13/Oct/20 16:29
• Resolved: 13/Oct/20 16:29

Description

Spark Version: 3.0.1

Description: The 'Shuffle Read Size / Records' field in Stage Summary metrics was shown as 'NaN Undefind'  when the readBytes value is negative,  as the attachment shows.

curl http:/hadoop001:18081/api/v1/applications/application_1601774913550_0225/stages/2/0/taskSummary

{
"quantiles" : [ 0.05, 0.25, 0.5, 0.75, 0.95 ],
"executorDeserializeTime" : [ 7.0, 357.0, 390.0, 484.0, 492.0 ],
 ...
"shuffleReadMetrics" :

{ "readBytes" : [ -2.0, 1775984.0, 1779759.0, 1781727.0, 1788426.0 ], "readRecords" : [ 2001.0, 2002.0, 2002.0, 2002.0, 2002.0 ],  ... }

Issue Links

• links to: [Github] Pull Request #30030 (akiyamaneko)