Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
None
-
None
-
None
Description
This is a slightly more complex version of SPARK-942 where we try to avoid unrolling iterators in other situations where it is possible. SPARK-942 focused on the case where the DISK_ONLY storage level was used. There are other cases though, such as if data is stored serialized and in memory and but there is not enough memory left to store the RDD.
Attachments
Issue Links
- is related to
-
SPARK-1777 Pass "cached" blocks directly to disk if memory is not large enough
- Resolved
- relates to
-
SPARK-942 Do not materialize partitions when DISK_ONLY storage level is used
- Resolved