[PARQUET-2238] Spec and parquet-mr disagree on DELTA_BYTE_ARRAY encoding

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Component/s: parquet-format, parquet-mr
• Assignee: Unassigned
• Reporter: Jan Finis
• Created: 04/Feb/23 11:30
• Updated: 23/Jun/24 03:32
• Resolved: 26/Sep/23 12:35

Description

The spec in parquet-format specifies that DELTA_BYTE_ARRAY is only supported for the physical type BYTE_ARRAY. Yet, parquet-mr also uses it to encode FIXED_LEN_BYTE_ARRAY.

So, I guess the spec should be updated to include FIXED_LEN_BYTE_ARRAY in the supported types of DELTA_BYTE_ARRAY encoding, or the code should be changed to no longer write this encoding for FIXED_LEN_BYTE_ARRAY.

I guess changing the spec is more prudent, given that 
a) the encoding can make sense for FIXED_LEN_BYTE_ARRAY
and
b) there might already be countless files written with this encoding / type combination.

Issue Links

• is duplicated by: PARQUET-2231 [Format] Encoding spec incorrect for DELTA_BYTE_ARRAY