[SPARK-16420] UnsafeShuffleWriter leaks compression streams with off-heap memory.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: Spark Core
• Assignee: Ryan Blue
• Reporter: Ryan Blue
• Created: 07/Jul/16 16:22
• Updated: 08/Jul/16 19:37
• Resolved: 08/Jul/16 19:37

Description

When merging spill files using Java file streams, UnsafeShuffleWriter opens a decompression stream for each shuffle part and never closes them. When the compression codec holds off-heap resources, these aren't cleaned up until the finalizer is called.

Issue Links

• links to: [Github] Pull Request #14093 (rdblue)