[SPARK-50235] Clean up ColumnVector resource after processing all rows in ColumnarToRowExec

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0, 3.4.4, 3.5.3
• Fix Version/s: 4.0.0, 3.5.4
• Component/s: SQL
• Labels: pull-request-available
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 05/Nov/24 22:13
• Updated: 1 week ago 14:47
• Resolved: 1 week ago 09:13

Description

Currently we only assign null to ColumnarBatch object but it doesn't release the resources hold by the vectors in the batch. For OnHeapColumnVector, the Java arrays may be automatically collected by JVM, but for OffHeapColumnVector, the allocated off-heap memory will be leaked.

For custom ColumnVector implementations like Arrow-based, it also possibly causes issues on memory safety if the underlying buffers are reused across batches. Because when ColumnarToRowExec begins to fill values for next batch, the arrays in previous batch are still hold.

Issue Links

• links to: GitHub Pull Request #48767