[SPARK-19937] Collect metrics of block sizes when shuffle.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.3.0
• Component/s: Shuffle, Spark Core
• Assignee: Jin Xing
• Reporter: Jin Xing
• Created: 13/Mar/17 13:34
• Updated: 17/May/20 18:31
• Resolved: 22/Jun/17 21:11

Description

This improvement is related to SPARK-19659 and also documented there. Metrics of blocks sizes(when shuffle) should be collected for later analysis. This is helpful for analysis when skew situations or OOM happens(though maxBytesInFlight is set).

Issue Links

• blocks: SPARK-19659 Fetch big blocks to disk when shuffle-read
• links to: [Github] Pull Request #17276 (jinxing64)
• links to: [Github] Pull Request #18249 (jinxing64)