-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.0.0
-
Fix Version/s: 2.0.0
-
Component/s: Spark Core
-
Labels:None
When merging spill files using Java file streams, UnsafeShuffleWriter opens a decompression stream for each shuffle part and never closes them. When the compression codec holds off-heap resources, these aren't cleaned up until the finalizer is called.