[SPARK-36123] Parquet vectorized reader doesn't skip null values correctly

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Labels: correctness
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 13/Jul/21 20:10
• Updated: 14/Jul/21 10:14
• Resolved: 14/Jul/21 10:14

Description

Currently when Parquet column index is effective, the vectorized reader doesn't skip null values properly which will cause incorrect result.

Issue Links

• is caused by: SPARK-34859 Vectorized parquet reader needs synchronization among pages for column index
• links to: [Github] Pull Request #33330 (sunchao)
• links to: [Github] Pull Request #33330 (sunchao)