Description
In ALS iterations, we checkpoint RDDs to cut lineage and to reduce shuffle files. However, whether to clean shuffle files depends on the system GC, which may not be triggered in ALS iterations. So after checkpointing, before we let the RDD object go out of scope, we should clean its shuffle dependencies explicitly. This function could either stay inside ALS or go to Core.
Without this feature, we can call System.gc() periodically to clean shuffle files of RDDs that went out of scope.
Attachments
Issue Links
- contains
-
SPARK-6334 spark-local dir not getting cleared during ALS
-
- Resolved
-
- is related to
-
SPARK-15447 Performance test for ALS in Spark 2.0
-
- Resolved
-
- links to
1.
|
Cleanup shuffles in ML version of ALS |
|
Closed | Unassigned |