[HIVE-21987] Hive is unable to read Parquet int32 annotated with decimal

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Assignee: Marta Kuczora
• Reporter: Nándor Kollár
• Created: 11/Jul/19 15:29
• Updated: 17/Nov/22 08:50
• Resolved: 01/Oct/19 11:32

Description

When I tried to read a Parquet file from a Hive (with Tez execution engine) table with a small decimal column, I got the following exception:

Caused by: java.lang.UnsupportedOperationException: org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$1
	at org.apache.parquet.io.api.PrimitiveConverter.addInt(PrimitiveConverter.java:98)
	at org.apache.parquet.column.impl.ColumnReaderImpl$2$3.writeValue(ColumnReaderImpl.java:248)
	at org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:367)
	at org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
	at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:226)
	... 28 more

Steps to reproduce:

• Create a Hive table with a single decimal(4, 2) column
• Create a Parquet file with int32 column annotated with decimal(4, 2) logical type, put it into the previously created table location (or use the attached parquet file, in this case the column should be named as 'd', to match the Hive schema with the Parquet schema in the file)
• Execute a select * on this table

Also, I'm afraid that similar problems can happen with int64 decimals too. Parquet specification allows both of these cases.

Issue Links

• duplicates: HIVE-12069 Hive cannot read Parquet decimals backed by INT32 or INT64
• duplicates: HIVE-19069 Hive can't read int32 and int64 Parquet decimal values
• links to: Review