[SPARK-38891] Skipping allocating vector for repetition & definition levels when possible

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0, 3.4.0
• Component/s: SQL
• Target Version/s: 3.3.0
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 13/Apr/22 19:49
• Updated: 16/May/22 12:36
• Resolved: 04/May/22 16:37

Description

Currently the vectorized Parquet reader will allocate vectors for repetition and definition levels in all cases. However in certain cases (e.g., when reading primitive types) this is not necessary and should be avoided.

Issue Links

• links to: [Github] Pull Request #36202 (sunchao)
• links to: [Github] Pull Request #36202 (sunchao)