[SPARK-40128] Add DELTA_LENGTH_BYTE_ARRAY as a recognized standalone encoding in VectorizedColumnReader

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0
• Component/s: SQL

Docs Text

Added support for keeping vectorized reads enabled for Parquet files using the DELTA_LENGTH_BYTE_ARRAY encoding as a standalone column encoding. Previously, the related DELTA_BINARY_PACKED and DELTA_BYTE_ARRAY encodings were accepted as column encodings, but DELTA_LENGTH_BYTE_ARRAY would still be rejected as "unsupported".

• Assignee: Dennis Huo
• Reporter: Dennis Huo
• Created: 17/Aug/22 17:13
• Updated: 17/Aug/22 23:38
• Resolved: 17/Aug/22 23:22

Description

Even though https://issues.apache.org/jira/browse/SPARK-36879 added implementations for DELTA_BINARY_PACKED, DELTA_BYTE_ARRAY, and DELTA_LENGTH_BYTE_ARRAY encodings, only DELTA_BINARY_PACKED and DELTA_BYTE_ARRAY were added as top-level standalone column encodings, with DELTA_LENGTH_BYTE_ARRAY only being used in its capacity as a subcomponent of DELTA_BYTE_ARRAY (for the non-shared string/binary suffixes).

Even though there apparently aren't many writers of the standalone DELTA_LENGTH_BYTE_ARRAY encoding, it's part of the core Parquet spec: https://parquet.apache.org/docs/file-format/data-pages/encodings/#delta-length-byte-array-delta_length_byte_array–6 and could be more efficient for types of binary/string data that don't take good advantage of sharing common prefixes for incremental encoding.

The problem can be reproduced by trying to load one of the https://github.com/apache/parquet-testing files (delta_length_byte_array.parquet).

Issue Links

• relates to: SPARK-36879 Support Parquet v2 data page encodings for the vectorized path
• links to: [Github] Pull Request #37557 (sfc-gh-dhuo)
• links to: [Github] Pull Request #37557 (sfc-gh-dhuo)