[SPARK-47910] Memory leak when interrupting shuffle write using zstd compression

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0, 3.3.0, 3.4.0, 3.5.0
• Fix Version/s: 4.0.0, 3.5.2
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: JacobZheng
• Reporter: JacobZheng
• Created: 19/Apr/24 05:58
• Updated: 18/Jun/24 05:07
• Resolved: 18/Jun/24 05:06

Description

When spark.sql.execution.interruptOnCancel=true and spark.io.compression.codec=zstd, a memory leak was found when tasks were cancelled at specific times. The reason for this is that cancelling a task interrupts the shuffle write, which then calls org.apache.spark.storage.DiskBlockObjectWriter#closeResources. this process then only closes the ManualCloseOutputStream, which is wrapped with this ZstdInputStreamNoFinalizer will not be closed. Moreover, ZstdInputStreamNoFinalizer doesn't implement Finalizer so it won't be reclaimed by GC automatically. 
 

Issue Links

• links to: GitHub Pull Request #46131