Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8205

Using strings in group type fails in ParquetSerDe

    XMLWordPrintableJSON

Details

    Description

      In HIVE-7735, schema info was plumbed to ETypeConverter to disambiguate between hive Char, Varchar and String types, which are all represented as PrimitiveType "binary" and OriginalType "utf8" in parquet.

      However, this does not work for parquet nested types (that map to hive Array, Map, etc.) containing these values, because schema lookup for nested values was not implemented. It's also non-trivial to do that in the current parquet serde implementation. Instead of plumbing in the schema, we should convert
      these types to the same Text writeable and let the object inspectors handle the final conversion.

      Attachments

        1. HIVE-8205.1.patch
          30 kB
          Xuefu Zhang
        2. HIVE-8205.1.patch
          30 kB
          Mohit Sabharwal
        3. HIVE-8205.patch
          29 kB
          Mohit Sabharwal

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HIVE-9371 Execution error for Parquet table and GROUP BY involving CHAR data type

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HIVE-7735 Implement Char, Varchar in ParquetSerDe

          • Major - Major loss of function.
          • Closed
          links to

          Web Link review board

          Activity

            People

              mohitsabharwal Mohit Sabharwal
              mohitsabharwal Mohit Sabharwal
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: