[HIVE-21215] Read Parquet INT64 timestamp

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Assignee: Marta Kuczora
• Reporter: Karen Coppage
• Created: 05/Feb/19 14:58
• Updated: 17/Nov/22 08:54
• Resolved: 04/Feb/20 10:05

Description

This patch enables Hive to start reading timestamps from Parquet written with the new semantics:

With Parquet version 1.11, a new timestamp LogicalType with base INT64 and the following metadata is introduced:

• boolean isAdjustedToUtc: marks whether the timestamp is converted to UTC (aka Instant semantics) or not (LocalDateTime semantics).
• enum TimeUnit (NANOS, MICROS, MILLIS): granularity of timestamp

Upon reading, the semantics of these new timestamps will be determined by their metadata, while the semantics of INT96 timestamps will continue to be deduced from the writer metadata.
This feature will be behind a flag for now.

Issue Links

• blocks: HIVE-21216 Write Parquet INT64 timestamp
• causes: HIVE-26612 INT64 Parquet timestamps cannot be read into BIGINT Hive type
• causes: HIVE-26658 INT64 Parquet timestamps cannot be mapped to most Hive numeric types
• duplicates: HIVE-13435 Hive fails to read timestamp stored as binary / int64 from externally generated parquet files
• is blocked by: HIVE-21050 Use Parquet LogicalTypes