Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0-alpha-1
    • None
    • None

    Description

      This patch enables Hive to start reading timestamps from Parquet written with the new semantics:

      With Parquet version 1.11, a new timestamp LogicalType with base INT64 and the following metadata is introduced:

      • boolean isAdjustedToUtc: marks whether the timestamp is converted to UTC (aka Instant semantics) or not (LocalDateTime semantics).
      • enum TimeUnit (NANOS, MICROS, MILLIS): granularity of timestamp

      Upon reading, the semantics of these new timestamps will be determined by their metadata, while the semantics of INT96 timestamps will continue to be deduced from the writer metadata.
      This feature will be behind a flag for now.

      Attachments

        1. HIVE-21215.1.patch
          18 kB
          Marta Kuczora
        2. HIVE-21215.2.patch
          19 kB
          Marta Kuczora

        Issue Links

          Activity

            zhxjdwh zhangxj added a comment - - edited

            Is this issue solved ? thanks ! klcopp kuczoram

            zhxjdwh zhangxj added a comment - - edited Is this issue solved ? thanks !  klcopp kuczoram
            kuczoram Marta Kuczora added a comment -

            Hi zhxjdwh, no this issue is not solved yet. The Parquet update is blocked by a recently found bug in ParquetFooterInputFromCache. (HIVE-22716). As soon as I get that fix in, we can go forward with upgrading the Parquet version and then with this patch.

            kuczoram Marta Kuczora added a comment - Hi zhxjdwh , no this issue is not solved yet. The Parquet update is blocked by a recently found bug in ParquetFooterInputFromCache. ( HIVE-22716 ). As soon as I get that fix in, we can go forward with upgrading the Parquet version and then with this patch.
            hiveqa Hive QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
                  master Compile Tests
            0 mvndep 2m 10s Maven dependency ordering for branch
            +1 mvninstall 7m 59s master passed
            +1 compile 1m 32s master passed
            +1 checkstyle 0m 53s master passed
            0 findbugs 0m 34s common in master has 63 extant Findbugs warnings.
            0 findbugs 4m 23s ql in master has 1537 extant Findbugs warnings.
            +1 javadoc 1m 18s master passed
                  Patch Compile Tests
            0 mvndep 0m 28s Maven dependency ordering for patch
            +1 mvninstall 1m 51s the patch passed
            +1 compile 1m 29s the patch passed
            +1 javac 1m 29s the patch passed
            -1 checkstyle 0m 44s ql: The patch generated 5 new + 18 unchanged - 1 fixed = 23 total (was 19)
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 findbugs 5m 13s the patch passed
            +1 javadoc 1m 19s the patch passed
                  Other Tests
            +1 asflicense 0m 15s The patch does not generate ASF License warnings.
            31m 6s



            Subsystem Report/Notes
            Optional Tests asflicense javac javadoc findbugs checkstyle compile
            uname Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux
            Build tool maven
            Personality /data/hiveptest/working/yetus_PreCommit-HIVE-Build-20435/dev-support/hive-personality.sh
            git revision master / d4248dc
            Default Java 1.8.0_111
            findbugs v3.0.1
            checkstyle http://104.198.109.242/logs//PreCommit-HIVE-Build-20435/yetus/diff-checkstyle-ql.txt
            modules C: common ql U: .
            Console output http://104.198.109.242/logs//PreCommit-HIVE-Build-20435/yetus.txt
            Powered by Apache Yetus http://yetus.apache.org

            This message was automatically generated.

            hiveqa Hive QA added a comment - -1 overall Vote Subsystem Runtime Comment       Prechecks +1 @author 0m 0s The patch does not contain any @author tags.       master Compile Tests 0 mvndep 2m 10s Maven dependency ordering for branch +1 mvninstall 7m 59s master passed +1 compile 1m 32s master passed +1 checkstyle 0m 53s master passed 0 findbugs 0m 34s common in master has 63 extant Findbugs warnings. 0 findbugs 4m 23s ql in master has 1537 extant Findbugs warnings. +1 javadoc 1m 18s master passed       Patch Compile Tests 0 mvndep 0m 28s Maven dependency ordering for patch +1 mvninstall 1m 51s the patch passed +1 compile 1m 29s the patch passed +1 javac 1m 29s the patch passed -1 checkstyle 0m 44s ql: The patch generated 5 new + 18 unchanged - 1 fixed = 23 total (was 19) +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 13s the patch passed +1 javadoc 1m 19s the patch passed       Other Tests +1 asflicense 0m 15s The patch does not generate ASF License warnings. 31m 6s Subsystem Report/Notes Optional Tests asflicense javac javadoc findbugs checkstyle compile uname Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux Build tool maven Personality /data/hiveptest/working/yetus_PreCommit-HIVE-Build-20435/dev-support/hive-personality.sh git revision master / d4248dc Default Java 1.8.0_111 findbugs v3.0.1 checkstyle http://104.198.109.242/logs//PreCommit-HIVE-Build-20435/yetus/diff-checkstyle-ql.txt modules C: common ql U: . Console output http://104.198.109.242/logs//PreCommit-HIVE-Build-20435/yetus.txt Powered by Apache Yetus http://yetus.apache.org This message was automatically generated.
            hiveqa Hive QA added a comment -

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12992361/HIVE-21215.1.patch

            SUCCESS: +1 due to 1 test(s) being added or modified.

            SUCCESS: +1 due to 17962 tests passed

            Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20435/testReport
            Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20435/console
            Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20435/

            Messages:

            Executing org.apache.hive.ptest.execution.TestCheckPhase
            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.YetusPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            

            This message is automatically generated.

            ATTACHMENT ID: 12992361 - PreCommit-HIVE-Build

            hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12992361/HIVE-21215.1.patch SUCCESS: +1 due to 1 test(s) being added or modified. SUCCESS: +1 due to 17962 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20435/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20435/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20435/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12992361 - PreCommit-HIVE-Build
            hiveqa Hive QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
                  master Compile Tests
            0 mvndep 2m 2s Maven dependency ordering for branch
            +1 mvninstall 7m 50s master passed
            +1 compile 1m 30s master passed
            +1 checkstyle 0m 52s master passed
            0 findbugs 0m 35s common in master has 63 extant Findbugs warnings.
            0 findbugs 4m 13s ql in master has 1537 extant Findbugs warnings.
            +1 javadoc 1m 18s master passed
                  Patch Compile Tests
            0 mvndep 0m 30s Maven dependency ordering for patch
            +1 mvninstall 1m 46s the patch passed
            +1 compile 1m 22s the patch passed
            +1 javac 1m 22s the patch passed
            -1 checkstyle 0m 41s ql: The patch generated 5 new + 18 unchanged - 1 fixed = 23 total (was 19)
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 findbugs 4m 59s the patch passed
            +1 javadoc 1m 16s the patch passed
                  Other Tests
            +1 asflicense 0m 15s The patch does not generate ASF License warnings.
            30m 8s



            Subsystem Report/Notes
            Optional Tests asflicense javac javadoc findbugs checkstyle compile
            uname Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux
            Build tool maven
            Personality /data/hiveptest/working/yetus_PreCommit-HIVE-Build-20443/dev-support/hive-personality.sh
            git revision master / 5acbffe
            Default Java 1.8.0_111
            findbugs v3.0.1
            checkstyle http://104.198.109.242/logs//PreCommit-HIVE-Build-20443/yetus/diff-checkstyle-ql.txt
            modules C: common ql U: .
            Console output http://104.198.109.242/logs//PreCommit-HIVE-Build-20443/yetus.txt
            Powered by Apache Yetus http://yetus.apache.org

            This message was automatically generated.

            hiveqa Hive QA added a comment - -1 overall Vote Subsystem Runtime Comment       Prechecks +1 @author 0m 0s The patch does not contain any @author tags.       master Compile Tests 0 mvndep 2m 2s Maven dependency ordering for branch +1 mvninstall 7m 50s master passed +1 compile 1m 30s master passed +1 checkstyle 0m 52s master passed 0 findbugs 0m 35s common in master has 63 extant Findbugs warnings. 0 findbugs 4m 13s ql in master has 1537 extant Findbugs warnings. +1 javadoc 1m 18s master passed       Patch Compile Tests 0 mvndep 0m 30s Maven dependency ordering for patch +1 mvninstall 1m 46s the patch passed +1 compile 1m 22s the patch passed +1 javac 1m 22s the patch passed -1 checkstyle 0m 41s ql: The patch generated 5 new + 18 unchanged - 1 fixed = 23 total (was 19) +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 59s the patch passed +1 javadoc 1m 16s the patch passed       Other Tests +1 asflicense 0m 15s The patch does not generate ASF License warnings. 30m 8s Subsystem Report/Notes Optional Tests asflicense javac javadoc findbugs checkstyle compile uname Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux Build tool maven Personality /data/hiveptest/working/yetus_PreCommit-HIVE-Build-20443/dev-support/hive-personality.sh git revision master / 5acbffe Default Java 1.8.0_111 findbugs v3.0.1 checkstyle http://104.198.109.242/logs//PreCommit-HIVE-Build-20443/yetus/diff-checkstyle-ql.txt modules C: common ql U: . Console output http://104.198.109.242/logs//PreCommit-HIVE-Build-20443/yetus.txt Powered by Apache Yetus http://yetus.apache.org This message was automatically generated.
            hiveqa Hive QA added a comment -

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12992488/HIVE-21215.2.patch

            SUCCESS: +1 due to 1 test(s) being added or modified.

            SUCCESS: +1 due to 17965 tests passed

            Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20443/testReport
            Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20443/console
            Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20443/

            Messages:

            Executing org.apache.hive.ptest.execution.TestCheckPhase
            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.YetusPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            

            This message is automatically generated.

            ATTACHMENT ID: 12992488 - PreCommit-HIVE-Build

            hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12992488/HIVE-21215.2.patch SUCCESS: +1 due to 1 test(s) being added or modified. SUCCESS: +1 due to 17965 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20443/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20443/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20443/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12992488 - PreCommit-HIVE-Build
            pvary Peter Vary added a comment -

            +1

            pvary Peter Vary added a comment - +1
            kuczoram Marta Kuczora added a comment -

            Pushed to master. Thanks a lot klcopp and pvary for the review.

            kuczoram Marta Kuczora added a comment - Pushed to master. Thanks a lot klcopp  and pvary  for the review.

            People

              kuczoram Marta Kuczora
              klcopp Karen Coppage
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: