Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-10666 Improvement for Parquet Predicate Push down on Hive
  3. HIVE-10642

Update Timestamp type mapping to Parquet

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      Timestamp in Hive was originally mapped to INT_96 in Parquet before. And inside Parquet it use Binary to save values.

      When working on HIVE-10255, I found predicate with timestamp does not work, since comparing Binary representative of Timestamp value can not give correct relationship.
      Meanwhile, in latest version Parquet, the timestamp is mapped to INT_64. Refer to https://github.com/Parquet/parquet-format/blob/master/LogicalTypes.md

      Therefore, this Jira will update the mapping of Timestamp type in Hive, and keep it backward compatible.

      Attachments

        1. HIVE-10642-parquet.patch
          11 kB
          Dong Chen
        2. HIVE-10642-parquet.1.patch
          12 kB
          Dong Chen

        Issue Links

          blocks

          Sub-task - The sub-task of the issue HIVE-10255 Parquet PPD support TIMESTAMP

          • Major - Major loss of function.
          • Patch Available
          duplicates

          Sub-task - The sub-task of the issue HIVE-10054 Clean up ETypeConverter since Parquet supports timestamp type already

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              dongc Dong Chen
              dongc Dong Chen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: