[SPARK-34816] Support for Parquet unsigned LogicalTypes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Kent Yao 2
• Reporter: Kent Yao 2
• Created: 22/Mar/21 03:19
• Updated: 26/Mar/21 05:36
• Resolved: 26/Mar/21 05:36

Description

Parquet supports some unsigned datatypes. Here is the definition related in parquet.thrift

/**
 * Common types used by frameworks(e.g. hive, pig) using parquet.  This helps map
 * between types in those frameworks to the base types in parquet.  This is only
 * metadata and not needed to read or write the data.
 */

  /**
   * An unsigned integer value.
   *
   * The number describes the maximum number of meaningful data bits in
   * the stored value. 8, 16 and 32 bit values are stored using the
   * INT32 physical type.  64 bit values are stored using the INT64
   * physical type.
   *
   */
  UINT_8 = 11;
  UINT_16 = 12;
  UINT_32 = 13;
  UINT_64 = 14;

Spark does not support unsigned datatypes. In SPARK-10113, we emit an exception with a clear message for them.

UInt8-[0:255]
UInt16-[0:65535]
UInt32-[0:4294967295]
UInt64-[0:18446744073709551615]

Unsigned types - may be used to produce smaller in-memory representations of the data. If the stored value is larger than the maximum allowed by int32 or int64, then the behavior is undefined.

In this ticket, we try to read them as a higher precision signed type

Issue Links

• is related to: SPARK-10113 Support for unsigned Parquet logical types

Sub-Tasks

• read parquet uint8/16/32 logical types: Resolved
• read parquet uint64 as decimal: Resolved