[SPARK-50463] Reading partition values from parquet over multiple batches fails

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0, 3.5.4
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Richard Chen
• Reporter: Richard Chen
• Created: 01/Dec/24 23:46
• Updated: 11/Dec/24 01:11
• Resolved: 03/Dec/24 16:37

Description

https://github.com/apache/spark/commit/800faf0abfa368ad0a5ef1e0fa44b74dbaab724e cleans up column vector resources in column to row. However, `ConstantColumnVector` holds the same value over multiple batches and shouldn't be cleaned up.

Issue Links

• links to: GitHub Pull Request #49021
• links to: GitHub Pull Request #49131