[SPARK-41096] Support reading parquet FIXED_LEN_BYTE_ARRAY type

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: Kazuyuki Tanimura
• Reporter: Kazuyuki Tanimura
• Created: 10/Nov/22 08:59
• Updated: 14/Nov/22 18:53
• Resolved: 14/Nov/22 18:53

Description

Parquet has FIXED_LEN_BYTE_ARRAY (FLBA) data type. However, Spark Parquet reader currently cannot handle it.
Read it as BinaryType in Spark.

Iceberg Parquet reader, for example, can handle FLBA. This improvement should reduce the gap between Spark and Iceberg Parquet reader.

Issue Links

• is related to: SPARK-2489 Unsupported parquet datatype optional fixed_len_byte_array
• links to: [Github] Pull Request #38628 (kazuyukitanimura)