Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21987

Hive is unable to read Parquet int32 annotated with decimal

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0-alpha-1
    • None
    • None

    Description

      When I tried to read a Parquet file from a Hive (with Tez execution engine) table with a small decimal column, I got the following exception:

      Caused by: java.lang.UnsupportedOperationException: org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$1
      	at org.apache.parquet.io.api.PrimitiveConverter.addInt(PrimitiveConverter.java:98)
      	at org.apache.parquet.column.impl.ColumnReaderImpl$2$3.writeValue(ColumnReaderImpl.java:248)
      	at org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:367)
      	at org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
      	at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:226)
      	... 28 more
      

      Steps to reproduce:

      • Create a Hive table with a single decimal(4, 2) column
      • Create a Parquet file with int32 column annotated with decimal(4, 2) logical type, put it into the previously created table location (or use the attached parquet file, in this case the column should be named as 'd', to match the Hive schema with the Parquet schema in the file)
      • Execute a select * on this table

      Also, I'm afraid that similar problems can happen with int64 decimals too. Parquet specification allows both of these cases.

      Attachments

        1. part-00000-e5287735-8dcf-4dda-9c6e-4d5c98dc15f2-c000.snappy.parquet
          0.4 kB
          Nándor Kollár
        2. HIVE-21987.5.patch
          56 kB
          Marta Kuczora
        3. HIVE-21987.4.patch
          56 kB
          Marta Kuczora
        4. HIVE-21987.3.patch
          54 kB
          Marta Kuczora
        5. HIVE-21987.2.patch
          54 kB
          Marta Kuczora
        6. HIVE-21987.1.patch
          52 kB
          Marta Kuczora

        Issue Links

          Activity

            People

              kuczoram Marta Kuczora
              nkollar Nándor Kollár
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: