[IMPALA-7853] Add support to read int64 NANO timestamps to the parquet scanner

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Fix Version/s: Impala 3.2.0
• Component/s: Backend
• Labels: parquet
• Epic Link: Switch Parquet timestamp format to INT64
• Target Version: Impala 3.2.0
• Epic Color: ghx-label-8
• Assignee: Csaba Ringhofer
• Reporter: Csaba Ringhofer
• Created: 15/Nov/18 16:19
• Updated: 14/Mar/19 14:52
• Resolved: 15/Jan/19 15:11

Description

PARQUET-1387 added int64 timestamps with nanosecond precision.

As 64 bits are not enough to represent the whole 1400..9999 range of Impala timestamps, this new new type works with a limited range:
1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807 UTC

The benefit of the reduced range is that no validation is necessary during scanning, as every possible 64 bit value represents a valid timestamp in Impala. This may mean that this has the potential be the fastest way to store timestamps in Impala + Parquet.

Another way NANO differs from MICRO and MILLI is that NANO can be only described with new logical types in Parquet, it has no converted type equivalent.