-
Type:
Improvement
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.1.0
-
Fix Version/s: 2.3.0
-
Component/s: Shuffle, Spark Core
-
Labels:None
This improvement is related to SPARK-19659 and also documented there. Metrics of blocks sizes(when shuffle) should be collected for later analysis. This is helpful for analysis when skew situations or OOM happens(though maxBytesInFlight is set).
- blocks
-
SPARK-19659 Fetch big blocks to disk when shuffle-read
-
- Resolved
-
- links to