[SPARK-36935] Enhance ParquetSchemaConverter to capture Parquet repetition & definition level

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 06/Oct/21 00:33
• Updated: 06/Aug/22 14:03
• Resolved: 02/Nov/21 23:02

Description

In order to support complex type for Parquet vectorized reader, we'll need to capture the repetition & definition level information associated with Catalyst Spark type converted from Parquet MessageType.

Issue Links

• Blocked: SPARK-39997 ParquetSchemaConverter fails match schema by id
• links to: [Github] Pull Request #34199 (sunchao)
• links to: [Github] Pull Request #34199 (sunchao)