Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7853

Add support to read int64 NANO timestamps to the parquet scanner

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: None
    • Fix Version/s: Impala 3.2.0
    • Component/s: Backend
    • Labels:

      Description

      PARQUET-1387 added int64 timestamps with nanosecond precision.

      As 64 bits are not enough to represent the whole 1400..9999 range of Impala timestamps, this new new type works with a limited range:
      1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807 UTC

      The benefit of the reduced range is that no validation is necessary during scanning, as every possible 64 bit value represents a valid timestamp in Impala. This may mean that this has the potential be the fastest way to store timestamps in Impala + Parquet.

      Another way NANO differs from MICRO and MILLI is that NANO can be only described with new logical types in Parquet, it has no converted type equivalent.

        Attachments

          Activity

            People

            • Assignee:
              csringhofer Csaba Ringhofer
              Reporter:
              csringhofer Csaba Ringhofer
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: