This improvement is related to SPARK-19659 and also documented there. Metrics of blocks sizes(when shuffle) should be collected for later analysis. This is helpful for analysis when skew situations or OOM happens(though maxBytesInFlight is set).
- blocks
-
SPARK-19659 Fetch big blocks to disk when shuffle-read
-
- Resolved
-
- links to
