[IMPALA-2111] Parquet TIMESTAMP is in milliseconds

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Done
• Affects Version/s: Impala 2.2, Impala 2.3.0
• Component/s: Backend
• Labels: parquet sqoop
• Epic Link: Timestamp Compatibility
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Andrés Cordero
• Created: 01/Jul/15 15:15
• Updated: 26/Nov/18 17:21
• Resolved: 26/Nov/18 17:21

Description

Based on documentation:
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/impala_parquet.html#parquet_data_types_unique_1

INT96 is mapping to TIMESTAMP in Impala.
A Sqoop import from DATE, DATETIME and TIMESTAMP will map to INT64 annotated with TIMESTAMP_MILLIS (sqlType 91/93) as described in:
https://github.com/apache/parquet-format/blob/master/LogicalTypes.md

In Impala this appears as a BIGINT, which casting to TIMESTAMP will produce wrong results as the underlying value is in milliseconds, while the cast expects seconds.

Impala should recognize INT64 + OriginalType DATE/DATETIME/TIMESTAMP as TIMESTAMP in milliseconds.

Issue Links

• is blocked by: IMPALA-5050 Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS to the parquet scanner