Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.1.0
-
None
Description
This improvement is related to SPARK-19659 and also documented there. Metrics of blocks sizes(when shuffle) should be collected for later analysis. This is helpful for analysis when skew situations or OOM happens(though maxBytesInFlight is set).
Attachments
Issue Links
- blocks
-
SPARK-19659 Fetch big blocks to disk when shuffle-read
- Resolved
- links to