Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5137

Support Kudu UNIXTIME_MICROS as Impala TIMESTAMP

        Details

        • Type: New Feature
        • Status: Resolved
        • Priority: Critical
        • Resolution: Fixed
        • Affects Version/s: Impala 2.8.0
        • Fix Version/s: Impala 2.9.0
        • Component/s: Backend
        • Labels:

          Description

          Impala aims to support TIMESTAMP for Kudu tables. Because Impala’s TIMESTAMP type is a 96-bit type with nanosecond precision and Kudu’s timestamp is a 64-bit microsecond delta from the Unix epoch (called UNIXTIME_MICROS), a conversion will be necessary.

            Activity

            Hide
            Permalink
            mjacobs Matthew Jacobs added a comment -

            main patch has been committed:
            commit a16a0fa84d77f96e428b278f8cc37ebd7a49899f
            Author: Matthew Jacobs <mj@cloudera.com>
            Date: Tue Mar 28 19:05:03 2017 -0700

            IMPALA-5137: Support Kudu UNIXTIME_MICROS as Impala TIMESTAMP

            Adds Impala support for TIMESTAMP types stored in Kudu.

            Impala stores TIMESTAMP values in 96-bits and has nanosecond
            precision. Kudu's timestamp is a 64-bit microsecond delta
            from the Unix epoch (called UNIXTIME_MICROS), so a conversion
            is necessary.

            When writing to Kudu, TIMESTAMP values in nanoseconds are
            averaged to the nearest microsecond.

            When reading from Kudu, the KuduScanner returns
            UNIXTIME_MICROS with 8bytes of padding so Impala can convert
            the value to a TimestampValue in-line and copy the entire
            row.

            Testing:
            Updated the functional_kudu schema to use TIMESTAMPs instead
            of converting to STRING, so this provides some decent
            coverage. Some BE tests were added, and some EE tests as
            well.

            TODO: Support pushing down TIMESTAMP predicates
            TODO: Support TIMESTAMPs in range partitioning expressions

            Change-Id: Iae6ccfffb79118a9036fb2227dba3a55356c896d
            Reviewed-on: http://gerrit.cloudera.org:8080/6526
            Reviewed-by: Matthew Jacobs <mj@cloudera.com>
            Tested-by: Impala Public Jenkins

            Show
            mjacobs Matthew Jacobs added a comment - main patch has been committed: commit a16a0fa84d77f96e428b278f8cc37ebd7a49899f Author: Matthew Jacobs <mj@cloudera.com> Date: Tue Mar 28 19:05:03 2017 -0700 IMPALA-5137 : Support Kudu UNIXTIME_MICROS as Impala TIMESTAMP Adds Impala support for TIMESTAMP types stored in Kudu. Impala stores TIMESTAMP values in 96-bits and has nanosecond precision. Kudu's timestamp is a 64-bit microsecond delta from the Unix epoch (called UNIXTIME_MICROS), so a conversion is necessary. When writing to Kudu, TIMESTAMP values in nanoseconds are averaged to the nearest microsecond. When reading from Kudu, the KuduScanner returns UNIXTIME_MICROS with 8bytes of padding so Impala can convert the value to a TimestampValue in-line and copy the entire row. Testing: Updated the functional_kudu schema to use TIMESTAMPs instead of converting to STRING, so this provides some decent coverage. Some BE tests were added, and some EE tests as well. TODO: Support pushing down TIMESTAMP predicates TODO: Support TIMESTAMPs in range partitioning expressions Change-Id: Iae6ccfffb79118a9036fb2227dba3a55356c896d Reviewed-on: http://gerrit.cloudera.org:8080/6526 Reviewed-by: Matthew Jacobs <mj@cloudera.com> Tested-by: Impala Public Jenkins

              People

              • Assignee:
                mjacobs Matthew Jacobs
                Reporter:
                mjacobs Matthew Jacobs
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development