[SPARK-50274] DirectByteBufferOutputStream may crash if used after close

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Ankur Dave
• Reporter: Ankur Dave
• Created: 08/Nov/24 23:29
• Updated: 10/Nov/24 09:40
• Resolved: 10/Nov/24 09:40

Description

`DirectByteBufferOutputStream#close()` calls `StorageUtils.dispose()` to free its direct byte buffer. This puts the object into an unspecified and dangerous state after being closed, and can cause unpredictable JVM crashes if it the object is used after close.

We can make this safer by modifying `close()` to place the object into a known-closed state, and modify all methods to assert not closed.

Issue Links

• links to: GitHub Pull Request #48807