[PARQUET-2420] ThriftParquetWriter converts thrift byte to int32 without adding logical type

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Component/s: parquet-thrift
• Assignee: Shreyas B
• Reporter: Shreyas B
• Created: 15/Jan/24 07:24
• Updated: 23/Jun/24 03:33
• Resolved: 19/Jan/24 14:21

Description

The current implementation of Parquet serialisation from Thrift Definitions results in the incorrect conversion of Thrift byte fields into INT32 without preserving the required LogicalType Metadata in the Parquet file. This behaviour leads to a loss of information and is inconsistent with the expected behaviour. The correct conversion should result in INT32 with LogicalType metadata indicating a bit width of 8 and signed as true.

 

Thrift Definition

struct TestLogicalType {
1: required i16 test_i16,
2: required byte test_i8
} 

Current Parquet Schema Generated

message ParquetSchema {
  required int32 test_i16 (INTEGER(16,true)) = 1;
  required int32 test_i8 = 2;
} 

Expected Parquet Schema 

message ParquetSchema {
  required int32 test_i16 (INTEGER(16,true)) = 1;
  required int32 test_i8 (INTEGER(8,true)) = 2;
} 

Issue Links

• links to: GitHub Pull Request #1254