Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12767

Implement table property to address Parquet int96 timestamp bug

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 1.2.1, 2.0.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Parque timestamps using INT96 are not compatible with other tools, like Impala, due to issues in Hive because it adjusts timezones values in a different way than Impala.

      To address such issues. a new table property (parquet.mr.int96.write.zone) must be used in Hive that detects what timezone to use when writing and reading timestamps from Parquet.

      The following is the exit criteria for the fix:

      • Hive will read Parquet MR int96 timestamp data and adjust values using a time zone from a table property, if set, or using the local time zone if it is absent. No adjustment will be applied to data written by Impala.
      • Hive will write Parquet int96 timestamps using a time zone adjustment from the same table property, if set, or using the local time zone if it is absent. This keeps the data in the table consistent.
      • New tables created by Hive will set the table property to UTC if the global option to set the property for new tables is enabled.
        • Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set the property unless the global setting to do so is enabled.
        • Tables created using CREATE TABLE LIKE <OTHER TABLE> will copy the property of the table that is copied.

        Attachments

        1. HIVE-12767.10.patch
          107 kB
          Barna Zsombor Klara
        2. HIVE-12767.11.patch
          121 kB
          Barna Zsombor Klara
        3. HIVE-12767.3.patch
          80 kB
          Sergio Peña
        4. HIVE-12767.4.patch
          80 kB
          Sergio Peña
        5. HIVE-12767.5.patch
          97 kB
          Barna Zsombor Klara
        6. HIVE-12767.6.patch
          99 kB
          Barna Zsombor Klara
        7. HIVE-12767.7.patch
          100 kB
          Barna Zsombor Klara
        8. HIVE-12767.8.patch
          107 kB
          Barna Zsombor Klara
        9. HIVE-12767.9.patch
          107 kB
          Barna Zsombor Klara
        10. TestNanoTimeUtils.java
          6 kB
          Ryan Blue

          Issue Links

            Activity

              People

              • Assignee:
                zsombor.klara Barna Zsombor Klara
                Reporter:
                spena Sergio Peña
              • Votes:
                1 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: